Nov 03 14:10:50.052527 ip-10-0-128-190 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 14:10:50.458084 ip-10-0-128-190 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 14:10:50.458084 ip-10-0-128-190 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 14:10:50.458084 ip-10-0-128-190 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 14:10:50.458084 ip-10-0-128-190 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 14:10:50.463824 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463745    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.463824 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463824    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.463824 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463828    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463832    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463836    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463838    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463841    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463845    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463847    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463850    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463853    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463855    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463858    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463861    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463864    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463866    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463876    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463881    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463885    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463888    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463891    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.463915 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463894    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463897    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463900    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463903    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463905    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463908    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463911    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463913    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463916    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463919    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463922    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463925    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463928    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463931    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463934    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463936    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463939    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463943    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463947    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.464399 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463950    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463953    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463955    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463958    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463960    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463966    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463969    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463971    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463974    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463977    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463979    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463982    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463984    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463987    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463989    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463993    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463995    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.463998    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464001    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464004    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.465086 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464007    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464009    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464013    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464016    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464019    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464021    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464024    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464026    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464029    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464031    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464034    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464037    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464040    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464042    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464045    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464047    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464050    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464052    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464056    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464059    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.465584 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464062    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464064    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464067    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464070    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464072    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464075    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464077    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464081    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464083    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464086    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464089    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464091    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464094    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464096    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464099    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464101    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464104    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464106    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464109    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.466053 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464111    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.466580 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464114    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.466580 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464116    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.466580 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464120    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.466580 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464123    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.466580 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464125    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.466580 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.464128    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.467388 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467374    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.467388 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467388    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467391    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467394    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467397    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467400    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467403    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467405    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467408    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467411    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467413    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467416    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467418    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467421    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467423    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467426    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467429    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467431    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467434    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467436    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.467449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467438    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467441    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467444    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467448    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467450    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467453    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467456    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467459    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467462    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467465    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467467    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467470    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467472    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467475    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467478    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467480    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467483    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467485    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467488    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.467914 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467490    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467493    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467496    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467498    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467501    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467503    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467506    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467509    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467512    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467514    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467517    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467519    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467522    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467526    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467529    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467533    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467536    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467539    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467542    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467545    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.468394 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467548    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467550    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467553    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467557    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467561    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467565    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467569    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467572    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467575    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467578    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467581    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467590    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467593    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467596    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467599    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467601    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467604    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467607    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467609    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.468874 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467612    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467615    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467618    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467621    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467623    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467627    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467630    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467633    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467635    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467638    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467641    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467643    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467646    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467648    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467651    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467653    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467656    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467660    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467662    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.469338 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467665    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467667    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467670    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467673    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467675    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467678    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467680    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467683    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.467685    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468184    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468188    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468192    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468195    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468198    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468201    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468204    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468206    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468209    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468213    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468216    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.469794 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468219    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468221    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468224    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468247    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468251    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468255    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468259    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468261    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468264    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468267    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468270    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468272    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468275    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468277    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468280    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468283    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468285    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468288    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468290    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468293    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.470300 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468295    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468298    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468300    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468304    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468307    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468310    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468312    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468315    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468318    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468320    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468323    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468326    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468329    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468332    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468335    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468337    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468340    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468342    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468345    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468347    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.470781 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468350    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468352    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468355    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468358    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468360    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468363    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468366    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468368    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468373    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468376    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468378    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468381    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468383    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468387    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468392    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468395    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468398    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468401    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468404    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.471267 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468407    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468409    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468418    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468421    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468423    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468426    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468429    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468432    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468435    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468438    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468440    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468443    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468445    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468448    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468450    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468454    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468458    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468462    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468465    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.471753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468468    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468470    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468473    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468476    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468479    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468481    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468484    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468486    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468489    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468491    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468494    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468498    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468500    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468503    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468506    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468509    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.468511    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470025    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470034    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470043    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470048    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 14:10:50.472198 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470052    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470056    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470063    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470067    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470071    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470074    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470077    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470080    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470084    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470087    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470089    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470093    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470095    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470099    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470103    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470106    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470109    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470112    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470116    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470120    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470123    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470126    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470129    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470132    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 14:10:50.472706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470136    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470139    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470143    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470146    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470150    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470153    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470156    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470159    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470162    2583 flags.go:64] FLAG: --enable-server="true"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470165    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470170    2583 flags.go:64] FLAG: --event-burst="100"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470174    2583 flags.go:64] FLAG: --event-qps="50"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470177    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470181    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470184    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470195    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470198    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470201    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470204    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470207    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470210    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470213    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470216    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470219    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470222    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 14:10:50.473377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470225    2583 flags.go:64] FLAG: --feature-gates=""
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470240    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470243    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470246    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470249    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470252    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470255    2583 flags.go:64] FLAG: --help="false"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470258    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470261    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470264    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470267    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470270    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470274    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470277    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470280    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470283    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470286    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470289    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470292    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470295    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470298    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470301    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470304    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470307    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 14:10:50.473983 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470310    2583 flags.go:64] FLAG: --lock-file=""
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470313    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470317    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470320    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470325    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470328    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470331    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470334    2583 flags.go:64] FLAG: --logging-format="text"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470337    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470340    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470343    2583 flags.go:64] FLAG: --manifest-url=""
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470346    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470351    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470354    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470358    2583 flags.go:64] FLAG: --max-pods="110"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470361    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470364    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470367    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470370    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470372    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470375    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470378    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470386    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470388    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470392    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 14:10:50.474621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470395    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470398    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470401    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470404    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470407    2583 flags.go:64] FLAG: --port="10250"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470410    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470413    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-053ceb1ac93318e17"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470416    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470419    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470426    2583 flags.go:64] FLAG: --register-node="true"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470429    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470433    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470436    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470438    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470441    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470445    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470448    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470451    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470454    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470457    2583 flags.go:64] FLAG: --runonce="false"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470460    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470463    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470466    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470469    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470472    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470475    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 14:10:50.475211 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470478    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470481    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470484    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470487    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470490    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470494    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470497    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470500    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470503    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470509    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470512    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470515    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470520    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470523    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470526    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470530    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470534    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470537    2583 flags.go:64] FLAG: --v="2"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470542    2583 flags.go:64] FLAG: --version="false"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470546    2583 flags.go:64] FLAG: --vmodule=""
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470550    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.470553    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470705    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470709    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.475839 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470713    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470716    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470719    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470722    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470725    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470729    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470732    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470736    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470739    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470742    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470745    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470748    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470751    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470754    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470757    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470760    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470763    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470765    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470768    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470771    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.476449 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470775    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470777    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470780    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470785    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470787    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470790    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470793    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470796    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470798    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470801    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470804    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470806    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470809    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470811    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470814    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470816    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470819    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470822    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470824    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.476934 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470827    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470830    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470832    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470835    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470837    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470840    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470842    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470845    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470848    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470850    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470854    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470856    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470859    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470863    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470865    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470868    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470872    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470875    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470877    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.477428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470880    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470882    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470885    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470887    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470890    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470892    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470895    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470897    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470900    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470902    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470905    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470907    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470910    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470913    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470915    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470917    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470920    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470923    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470925    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470928    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.477875 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470931    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470933    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470936    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470939    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470941    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470944    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470947    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470950    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470953    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470957    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470959    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470962    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470964    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470967    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470970    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470972    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470975    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470977    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470981    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.478357 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470985    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470988    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470991    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470994    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.470997    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471000    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471002    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.471029    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471065    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471070    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471073    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471076    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471079    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471082    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471085    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471088    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.478802 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471090    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471094    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471096    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471100    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471103    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471106    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471108    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471112    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471115    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471118    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471120    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471123    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471125    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471128    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471131    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471133    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471136    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471138    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471141    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.479199 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471144    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471147    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471149    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471152    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471154    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471157    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471159    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471162    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471164    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471167    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471171    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471176    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471179    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471182    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471185    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471188    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471192    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471195    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471197    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.479676 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471200    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471203    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471206    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471209    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471211    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471214    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471218    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471221    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471224    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471238    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471241    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471243    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471246    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471249    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471251    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471254    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471256    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471259    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471262    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471264    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.480123 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471267    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471270    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471272    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471275    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471278    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471281    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471284    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471286    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471289    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471293    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471296    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471299    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471301    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471305    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471307    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471310    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471312    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471315    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471317    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.480603 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471320    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471322    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471325    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471327    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471330    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471332    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471335    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471337    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471340    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471342    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471345    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471348    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471353    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471355    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471358    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471360    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471363    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471365    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471368    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471371    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.481047 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.471373    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.478033    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.478048    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.478932    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.478939    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.478994    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479000    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479004    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479006    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479009    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479012    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479015    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479018    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479020    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479023    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479025    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479028    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479031    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479034    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479037    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.481560 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479040    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479042    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479045    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479047    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479050    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479052    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479055    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479058    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479060    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479063    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479065    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479068    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479070    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479073    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479077    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479079    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479082    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479086    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479090    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.482066 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479093    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479096    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479098    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479101    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479104    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479107    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479109    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479112    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479115    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479117    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479120    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479122    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479125    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479132    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479135    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479138    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479140    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479143    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479145    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479148    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.482499 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479150    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479153    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479155    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479158    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479160    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479163    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479166    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479169    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479172    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479175    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479178    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479180    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479183    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479185    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479188    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479190    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479194    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479198    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479201    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.482960 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479205    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479207    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479210    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479213    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479215    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479218    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479221    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479223    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479238    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479242    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479246    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479250    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479253    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479255    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479258    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479260    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479263    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479266    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479269    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.483405 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479272    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479274    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479277    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479280    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479283    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479285    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479288    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479290    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479293    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479296    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479298    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479301    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479303    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479306    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.479336    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479491    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.483850 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479496    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479500    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479503    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479506    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479508    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479511    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479514    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479517    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479519    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479522    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479524    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479527    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479529    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479532    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479534    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479537    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479539    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479542    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479545    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479547    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.484219 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479550    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479553    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479555    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479558    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479560    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479563    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479565    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479568    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479571    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479573    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479575    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479578    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479580    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479589    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479593    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479596    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479599    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479602    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479604    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.484736 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479607    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479610    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479612    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479616    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479620    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479623    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479626    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479628    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479631    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479633    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479636    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479639    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479641    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479644    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479653    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479656    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479659    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479661    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479664    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479667    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.485180 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479669    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479672    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479674    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479677    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479679    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479682    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479685    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479687    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479690    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479693    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479695    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479698    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479700    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479703    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479705    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479708    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479710    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479713    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479715    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479718    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.485664 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479721    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479723    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479726    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479729    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479731    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479734    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479736    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479745    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479748    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479750    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479753    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479755    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479758    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479760    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479763    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479765    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479768    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479770    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479773    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479776    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.486137 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479778    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479781    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479784    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479788    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479791    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479794    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.479818    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479858    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479862    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479865    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479868    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479871    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479874    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479876    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479881    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:10:50.486670 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479884    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479886    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479889    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479892    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479894    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479902    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479905    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479908    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479911    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479913    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479916    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479918    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479921    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479923    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479926    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479928    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479931    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479934    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479937    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479939    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:10:50.487026 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479942    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479944    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479947    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479949    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479952    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479954    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479957    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479959    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479962    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479964    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479967    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479971    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479973    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479976    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479979    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479981    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479984    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479987    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479995    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:10:50.487615 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.479998    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480000    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480003    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480007    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480011    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480014    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480017    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480020    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480022    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480025    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480028    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480031    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480033    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480036    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480039    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480041    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480044    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480046    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480049    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:10:50.488169 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480052    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480054    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480057    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480059    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480062    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480066    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480069    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480071    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480074    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480076    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480079    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480082    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480084    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480092    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480095    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480098    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480100    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480103    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480106    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480108    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:10:50.488643 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480111    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480113    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480116    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480118    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480121    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480123    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480126    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480129    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480132    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480135    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480139    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480143    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480146    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480149    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480152    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480163    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480166    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480170    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480174    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:10:50.489100 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:50.480177    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.480244    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.482576    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.483679    2583 server.go:1014] "Starting client certificate rotation"
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.483787    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.484750    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.485960    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 14:10:50.489561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.487342    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 14:10:50.505990 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.505971    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 03 14:10:50.509582 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.509565    2583 log.go:25] "Validated CRI v1 image API"
Nov 03 14:10:50.510461 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.510438    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 14:10:50.514939 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.514919    2583 fs.go:135] Filesystem UUIDs: map[73e197d9-c4a4-4e87-b00e-64552d1e35da:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c6f67808-7d54-469f-8aa0-a28cb27b0a11:/dev/nvme0n1p3]
Nov 03 14:10:50.515004 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.514937    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 14:10:50.519872 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.519764    2583 manager.go:217] Machine: {Timestamp:2026-11-03 14:10:50.518495914 +0000 UTC m=+0.347740830 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124371 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f369d823326f19dc633f4229cbde SystemUUID:ec28f369-d823-326f-19dc-633f4229cbde BootID:bde9a5c6-48e4-43f8-a770-a3b57f8f45d5 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f4:bc:f6:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f4:bc:f6:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:f4:03:54:e2:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 14:10:50.520559 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.520548    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 14:10:50.520637 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.520625    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 14:10:50.524537 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.524509    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 14:10:50.524695 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.524540    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 14:10:50.524787 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.524700    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 14:10:50.524787 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.524713    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 14:10:50.524787 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.524746    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 14:10:50.525844 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.525831    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 14:10:50.526791 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.526778    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 14:10:50.530914 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.530900    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 14:10:50.531046 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.531034    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 14:10:50.531985 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.531974    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 14:10:50.532049 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.531992    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 14:10:50.532049 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.532006    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 14:10:50.532049 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.532020    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 03 14:10:50.532049 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.532042    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 14:10:50.532445 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.532430    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 14:10:50.536070 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.536046    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 14:10:50.543114 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.543101    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 14:10:50.544085 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544072    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 14:10:50.544122 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544104    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 14:10:50.544273 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544263    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 14:10:50.544302 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544279    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 14:10:50.544302 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544286    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 14:10:50.544302 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544292    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 14:10:50.544302 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544299    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 14:10:50.544405 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.544308    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 14:10:50.545010 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.545002    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 14:10:50.545040 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.545012    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 14:10:50.545040 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.545020    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 14:10:50.545040 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.545026    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 14:10:50.545040 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.545036    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 14:10:50.545154 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.545044    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 14:10:50.546793 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.546783    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 14:10:50.546834 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.546796    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 14:10:50.551349 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.551336    2583 server.go:1263] "Started kubelet"
Nov 03 14:10:50.551574 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.551463    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 14:10:50.551692 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.551680    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 14:10:50.552259 ip-10-0-128-190 systemd[1]: Started Kubernetes Kubelet.
Nov 03 14:10:50.556369 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.551423    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 14:10:50.558757 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.558728    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 14:10:50.559006 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.558989    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 14:10:50.563941 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.563927    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 14:10:50.564018 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.563943    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 14:10:50.564018 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564007    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 14:10:50.564125 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.564028    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:50.564766 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564745    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 14:10:50.564766 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564766    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 14:10:50.564895 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564866    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 14:10:50.564895 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564884    2583 factory.go:55] Registering systemd factory
Nov 03 14:10:50.564895 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564892    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 03 14:10:50.565034 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564907    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 14:10:50.565034 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564916    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 14:10:50.565034 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.564953    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 14:10:50.565174 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.565165    2583 factory.go:153] Registering CRI-O factory
Nov 03 14:10:50.565221 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.565179    2583 factory.go:223] Registration of the crio container factory successfully
Nov 03 14:10:50.565221 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.565193    2583 factory.go:103] Registering Raw factory
Nov 03 14:10:50.565221 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.565203    2583 manager.go:1199] Started watching for new ooms in manager
Nov 03 14:10:50.566254 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.566220    2583 manager.go:319] Starting recovery of all containers
Nov 03 14:10:50.579479 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.579339    2583 manager.go:324] Recovery completed
Nov 03 14:10:50.581351 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.581320    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 14:10:50.581482 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.581336    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 14:10:50.582647 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.581312    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c061e349  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.551305033 +0000 UTC m=+0.380549951,LastTimestamp:2026-11-03 14:10:50.551305033 +0000 UTC m=+0.380549951,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.583724 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.583700    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.585825 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.585809    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.585885 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.585836    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.585885 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.585846    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.586405 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.586387    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 03 14:10:50.586405 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.586399    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 14:10:50.586496 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.586416    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 14:10:50.586496 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.586399    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.587793 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.587779    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.587863 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.587799    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.587863 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.587808    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.588687 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.588669    2583 policy_none.go:50] "Start"
Nov 03 14:10:50.588687 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.588689    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 14:10:50.588823 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.588702    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 14:10:50.590523 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.590506    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 14:10:50.590853 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.590841    2583 policy_none.go:44] "Start"
Nov 03 14:10:50.596669 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.596641    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 14:10:50.596669 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.596640    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 14:10:50.596801 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.596668    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 14:10:50.609122 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.609058    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c270998c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.585823628 +0000 UTC m=+0.415068544,LastTimestamp:2026-11-03 14:10:50.585823628 +0000 UTC m=+0.415068544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.625699 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.625557    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 14:10:50.632213 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.632137    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c270dbe4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-190.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.585840612 +0000 UTC m=+0.415085528,LastTimestamp:2026-11-03 14:10:50.585840612 +0000 UTC m=+0.415085528,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.642846 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.642833    2583 manager.go:341] "Starting Device Plugin manager"
Nov 03 14:10:50.642913 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.642861    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 14:10:50.642913 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.642870    2583 server.go:87] "Starting device plugin registration server"
Nov 03 14:10:50.643107 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.643096    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 14:10:50.643141 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.643108    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 14:10:50.643256 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.643221    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 14:10:50.643462 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.643339    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 14:10:50.643462 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.643347    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 14:10:50.643855 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.643835    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 14:10:50.643921 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.643877    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:50.658670 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.658602    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c27103fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-190.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.585850875 +0000 UTC m=+0.415095790,LastTimestamp:2026-11-03 14:10:50.585850875 +0000 UTC m=+0.415095790,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.679678 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.679607    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-190.ec2.internal.18e48e55c270998c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c270998c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.585823628 +0000 UTC m=+0.415068544,LastTimestamp:2026-11-03 14:10:50.587791471 +0000 UTC m=+0.417036387,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.705260 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.705178    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-190.ec2.internal.18e48e55c270dbe4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c270dbe4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-190.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.585840612 +0000 UTC m=+0.415085528,LastTimestamp:2026-11-03 14:10:50.587803047 +0000 UTC m=+0.417047965,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.705969 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.705951    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-77lxj"
Nov 03 14:10:50.715053 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.715009    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-77lxj"
Nov 03 14:10:50.726776 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.726715    2583 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-190.ec2.internal.18e48e55c27103fb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-190.ec2.internal.18e48e55c27103fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-190.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-190.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-190.ec2.internal,},FirstTimestamp:2026-11-03 14:10:50.585850875 +0000 UTC m=+0.415095790,LastTimestamp:2026-11-03 14:10:50.58781209 +0000 UTC m=+0.417057006,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-190.ec2.internal,}"
Nov 03 14:10:50.743989 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.743969    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.744911 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.744898    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.744996 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.744927    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.744996 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.744942    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.744996 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.744977    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.786821 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.765711    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.786821 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.765731    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-190.ec2.internal\": node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:50.811571 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.811542    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 14:10:50.812725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.812710    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 14:10:50.812803 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.812731    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 14:10:50.812803 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.812747    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 14:10:50.812803 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.812787    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 14:10:50.826894 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.826877    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:50.843752 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.843730    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:10:50.913756 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.913734    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"]
Nov 03 14:10:50.913821 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.913788    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.914668 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.914652    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.914757 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.914682    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.914757 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.914704    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.916819 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.916804    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.916973 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.916960    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.917020 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.916988    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.917600 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.917585    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.917600 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.917594    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.917732 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.917615    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.917732 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.917615    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.917732 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.917641    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.917732 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.917627    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.919759 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.919742    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.919841 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.919772    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:10:50.920464 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.920449    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:10:50.920563 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.920473    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:10:50.920563 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.920484    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:10:50.927835 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.927821    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:50.951806 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.951786    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-190.ec2.internal\" not found" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.953866 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:50.953850    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-190.ec2.internal\" not found" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.966991 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.966947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.966991 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.966974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:50.967099 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:50.967000    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61a071342255b3898148bdccf890b4bc-config\") pod \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" (UID: \"61a071342255b3898148bdccf890b4bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.028546 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.028530    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.067740 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.067724    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.067832 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.067748    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.067832 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.067764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61a071342255b3898148bdccf890b4bc-config\") pod \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" (UID: \"61a071342255b3898148bdccf890b4bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.067832 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.067812    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61a071342255b3898148bdccf890b4bc-config\") pod \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" (UID: \"61a071342255b3898148bdccf890b4bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.067942 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.067835    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.067942 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.067834    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.128865 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.128846    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.229883 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.229838    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.256027 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.256012    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.258176 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.258162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.330116 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.330086    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.430740 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.430712    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.449383 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.449366    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:10:51.483539 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.483481    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 14:10:51.483960 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.483630    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 14:10:51.483960 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.483661    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 14:10:51.534155 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.534131    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.634247 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:51.634197    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:10:51.715629 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.715606    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:10:51.728741 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.728712    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 14:05:50 +0000 UTC" deadline="2028-07-28 16:37:14.27967768 +0000 UTC"
Nov 03 14:10:51.728741 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.728740    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15194h26m22.550941959s"
Nov 03 14:10:51.764500 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.764439    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.789760 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.789739    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 14:10:51.791665 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.791651    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:10:51.821444 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.821426    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 14:10:51.873274 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:51.873229    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a96a710dae74d27a1c0a4680122307.slice/crio-dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c WatchSource:0}: Error finding container dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c: Status 404 returned error can't find the container with id dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c
Nov 03 14:10:51.873790 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:51.873771    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a071342255b3898148bdccf890b4bc.slice/crio-096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b WatchSource:0}: Error finding container 096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b: Status 404 returned error can't find the container with id 096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b
Nov 03 14:10:51.879050 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:51.878978    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:10:52.208215 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.208191    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:10:52.534371 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.534292    2583 apiserver.go:52] "Watching apiserver"
Nov 03 14:10:52.565596 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.565556    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 14:10:52.568553 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.568525    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 14:10:52.568877 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.568855    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-k8qn5","openshift-dns/node-resolver-g9r7l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal","openshift-multus/multus-additional-cni-plugins-n6kkm","kube-system/global-pull-secret-syncer-8flw8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn","kube-system/konnectivity-agent-7t76q","openshift-multus/multus-9f6lw","openshift-multus/network-metrics-daemon-9vstd","openshift-ovn-kubernetes/ovnkube-node-f55sl","kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal","openshift-image-registry/node-ca-868d4","openshift-network-diagnostics/network-check-target-s7z9p","openshift-network-operator/iptables-alerter-kmcrp"]
Nov 03 14:10:52.571619 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.571589    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.575444 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.575423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.576679 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.576659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.578923 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.578898    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.581144 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.581125    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.581240 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.581208    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:10:52.583470 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.583448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.586591 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.586573    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.589050 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.589029    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.591192 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.591158    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trbj6\""
Nov 03 14:10:52.591472 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.591451    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 14:10:52.592906 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.592027    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:52.592906 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.592108    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:10:52.594646 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.594629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.597268    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.597999    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598132    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598134    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lqmd6\""
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598165    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95wl7\""
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598202    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q495v\""
Nov 03 14:10:52.598296 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598283    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598395    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598441    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598483    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598523    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598536    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598566    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598487    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.598701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.598626    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 14:10:52.599829 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.599809    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:52.599938 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.599884    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:10:52.629402 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.629377    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 14:10:52.629505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.629455    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-76vsv"
Nov 03 14:10:52.644027 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.644007    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-76vsv"
Nov 03 14:10:52.666480 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.666456    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 14:10:52.676680 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676656    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.676784 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.676784 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-var-lib-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.676784 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.676920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676796    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29be6-c48c-4bc8-a601-a0e337715b61-konnectivity-ca\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.676920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-system-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.676920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cnibin\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.676920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgnm\" (UniqueName: \"kubernetes.io/projected/40daba9a-1280-486d-b7f4-047d0efdab2e-kube-api-access-blgnm\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:52.677100 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dncp\" (UniqueName: \"kubernetes.io/projected/1eb511bd-1e7a-44c1-b027-964d3fb57d08-kube-api-access-2dncp\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.677100 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8edbb9a8-e48f-428f-9967-61dd8333133e-serviceca\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.677100 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.676984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.677100 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677006    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ns6\" (UniqueName: \"kubernetes.io/projected/9713304b-d13e-4e2c-b460-6a02c9adf5b4-kube-api-access-f9ns6\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.677100 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677030    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.677100 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.677390 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-var-lib-kubelet\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.677390 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677183    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cni-binary-copy\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.677390 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:52.677390 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677332    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb511bd-1e7a-44c1-b027-964d3fb57d08-host-slash\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.677390 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677385    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-kubelet\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.677592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtkpv\" (UniqueName: \"kubernetes.io/projected/8edbb9a8-e48f-428f-9967-61dd8333133e-kube-api-access-qtkpv\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.677592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677449    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c76fe7-8625-4a41-a039-af53c4978a94-tmp-dir\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.677592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677495    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-tmp\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.677592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.677761 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-socket-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.677761 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677643    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.677761 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-systemd\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.677761 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-sys\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.677928 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-hostroot\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.677928 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-daemon-config\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.677928 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1eb511bd-1e7a-44c1-b027-964d3fb57d08-iptables-alerter-script\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.677928 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677855    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-log-socket\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.677928 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovn-node-metrics-cert\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-dbus\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.677965    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cnibin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-registration-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-device-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-slash\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-etc-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-ovn\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678153    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-script-lib\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678195    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-multus\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678274    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpfvh\" (UniqueName: \"kubernetes.io/projected/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-kube-api-access-mpfvh\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-sys-fs\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-systemd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678365    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-netns\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678410    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-socket-dir-parent\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678501    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-config\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.678548 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-lib-modules\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-k8s-cni-cncf-io\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678654    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-conf-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-multus-certs\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1c76fe7-8625-4a41-a039-af53c4978a94-hosts-file\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-system-cni-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e29be6-c48c-4bc8-a601-a0e337715b61-agent-certs\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysconfig\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678898    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.678950 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678933    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-netns\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678964    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-env-overrides\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.678984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-modprobe-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679015    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-run\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-systemd-units\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-bin\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edbb9a8-e48f-428f-9967-61dd8333133e-host\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679112    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679156    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-tuned\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8729\" (UniqueName: \"kubernetes.io/projected/0977c82f-e896-4817-b4bf-eb1b056060c6-kube-api-access-d8729\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679199    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc45c\" (UniqueName: \"kubernetes.io/projected/a4fa7bf3-f448-451d-b46e-a885de36da57-kube-api-access-xc45c\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.679322 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679325    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-node-log\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679346    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr6tm\" (UniqueName: \"kubernetes.io/projected/b1c76fe7-8625-4a41-a039-af53c4978a94-kube-api-access-fr6tm\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-kubelet\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679379    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-netd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65rvz\" (UniqueName: \"kubernetes.io/projected/eb21c178-123e-4c5c-a0f3-e59e5aab8927-kube-api-access-65rvz\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679414    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-kubelet-config\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679488    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-kubernetes\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679515    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-conf\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-host\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679558    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-bin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.679725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.679582    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-etc-kubernetes\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.700748 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.700719    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.700748 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.700721    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tb44c\""
Nov 03 14:10:52.700941 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.700911    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.701032 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701016    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.701073 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701059    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.701119 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701108    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.701189 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701172    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 14:10:52.701189 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701184    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xm8pf\""
Nov 03 14:10:52.701400 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701251    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 14:10:52.701400 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701351    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2ft6v\""
Nov 03 14:10:52.701400 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701251    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhrll\""
Nov 03 14:10:52.701560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701023    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 14:10:52.701560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701463    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 14:10:52.701560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701490    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.701560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701516    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 14:10:52.701560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701549    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgxkz\""
Nov 03 14:10:52.701560 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701557    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 14:10:52.701858 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701586    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 14:10:52.701858 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701617    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 14:10:52.701858 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.701626    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 14:10:52.779803 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-conf\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.779803 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-host\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779847    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-bin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779926    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-bin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779928    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-host\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779951    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-etc-kubernetes\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779989    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-etc-kubernetes\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779991    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-conf\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.780029 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.779994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-var-lib-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780099    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780123    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29be6-c48c-4bc8-a601-a0e337715b61-konnectivity-ca\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780173    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-system-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780179    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-var-lib-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cnibin\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780209    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780260    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cnibin\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780267    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-system-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780302    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgnm\" (UniqueName: \"kubernetes.io/projected/40daba9a-1280-486d-b7f4-047d0efdab2e-kube-api-access-blgnm\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dncp\" (UniqueName: \"kubernetes.io/projected/1eb511bd-1e7a-44c1-b027-964d3fb57d08-kube-api-access-2dncp\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8edbb9a8-e48f-428f-9967-61dd8333133e-serviceca\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.780398 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780385    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ns6\" (UniqueName: \"kubernetes.io/projected/9713304b-d13e-4e2c-b460-6a02c9adf5b4-kube-api-access-f9ns6\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-var-lib-kubelet\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cni-binary-copy\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780543    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb511bd-1e7a-44c1-b027-964d3fb57d08-host-slash\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780607    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-kubelet\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780617    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-var-lib-kubelet\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkpv\" (UniqueName: \"kubernetes.io/projected/8edbb9a8-e48f-428f-9967-61dd8333133e-kube-api-access-qtkpv\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c76fe7-8625-4a41-a039-af53c4978a94-tmp-dir\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-tmp\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780733    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780738    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb511bd-1e7a-44c1-b027-964d3fb57d08-host-slash\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780753    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-socket-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780799    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-systemd\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.780841    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780853    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-sys\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780896    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-sys\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780903    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-hostroot\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780940    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-hostroot\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.780945    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:10:53.280899288 +0000 UTC m=+3.110144218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.780988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-daemon-config\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1eb511bd-1e7a-44c1-b027-964d3fb57d08-iptables-alerter-script\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781121    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c76fe7-8625-4a41-a039-af53c4978a94-tmp-dir\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781157    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-socket-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781174    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-systemd\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781188    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781204    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.781920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781215    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-kubelet\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781228    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-log-socket\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-log-socket\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovn-node-metrics-cert\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-dbus\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781353    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8edbb9a8-e48f-428f-9967-61dd8333133e-serviceca\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781358    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29be6-c48c-4bc8-a601-a0e337715b61-konnectivity-ca\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781369    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cnibin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-registration-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-device-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781442    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-slash\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781443    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-dbus\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-etc-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781492    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-ovn\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781504    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-registration-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-script-lib\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-multus\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781562    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cnibin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.782681 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781548    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781608    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-etc-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-device-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781624    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-ovn\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781631    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1eb511bd-1e7a-44c1-b027-964d3fb57d08-iptables-alerter-script\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781572    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781663    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-slash\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfvh\" (UniqueName: \"kubernetes.io/projected/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-kube-api-access-mpfvh\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781689    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781709    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-multus\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781772    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-sys-fs\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-systemd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781934    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-netns\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781955    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-socket-dir-parent\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.781993    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-systemd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.783592 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782011    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-config\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782052    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-sys-fs\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-lib-modules\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-k8s-cni-cncf-io\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782102    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-socket-dir-parent\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-conf-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782140    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-netns\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-multus-certs\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1c76fe7-8625-4a41-a039-af53c4978a94-hosts-file\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782177    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-daemon-config\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-system-cni-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782221    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e29be6-c48c-4bc8-a601-a0e337715b61-agent-certs\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.782225    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782244    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cni-binary-copy\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782244    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-script-lib\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.782293    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:53.282275188 +0000 UTC m=+3.111520110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:52.785526 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782299    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1c76fe7-8625-4a41-a039-af53c4978a94-hosts-file\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782235    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-k8s-cni-cncf-io\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782332    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-lib-modules\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782361    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-system-cni-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782367    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-conf-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782393    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-multus-certs\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysconfig\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782446    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysconfig\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782449    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-netns\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-env-overrides\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782515    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782541    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-modprobe-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782550    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-netns\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-run\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782610    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-systemd-units\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-bin\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.786505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edbb9a8-e48f-428f-9967-61dd8333133e-host\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782732    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-tuned\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782785    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8729\" (UniqueName: \"kubernetes.io/projected/0977c82f-e896-4817-b4bf-eb1b056060c6-kube-api-access-d8729\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc45c\" (UniqueName: \"kubernetes.io/projected/a4fa7bf3-f448-451d-b46e-a885de36da57-kube-api-access-xc45c\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782827    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-bin\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782825    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-node-log\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6tm\" (UniqueName: \"kubernetes.io/projected/b1c76fe7-8625-4a41-a039-af53c4978a94-kube-api-access-fr6tm\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782885    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-run\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782891    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-systemd-units\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-kubelet\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-netd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782940    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edbb9a8-e48f-428f-9967-61dd8333133e-host\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rvz\" (UniqueName: \"kubernetes.io/projected/eb21c178-123e-4c5c-a0f3-e59e5aab8927-kube-api-access-65rvz\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782977    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-modprobe-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-kubelet-config\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.787036 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782995    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-env-overrides\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.783017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-kubernetes\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.783072    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-node-log\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.783099    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-kubernetes\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.782784    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-config\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.783152    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-netd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.783293    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-kubelet-config\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.783354    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-kubelet\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.785880    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovn-node-metrics-cert\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.786343    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e29be6-c48c-4bc8-a601-a0e337715b61-agent-certs\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.786451    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-tmp\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.787608 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.786496    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-tuned\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.805262 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.805223    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:10:52.805262 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.805251    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:10:52.805262 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.805265    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:52.805517 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:52.805364    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:53.305342672 +0000 UTC m=+3.134587596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:52.806594 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.806567    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dncp\" (UniqueName: \"kubernetes.io/projected/1eb511bd-1e7a-44c1-b027-964d3fb57d08-kube-api-access-2dncp\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.807924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.807892    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkpv\" (UniqueName: \"kubernetes.io/projected/8edbb9a8-e48f-428f-9967-61dd8333133e-kube-api-access-qtkpv\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.807924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.807917    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6tm\" (UniqueName: \"kubernetes.io/projected/b1c76fe7-8625-4a41-a039-af53c4978a94-kube-api-access-fr6tm\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.809113 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.808955    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfvh\" (UniqueName: \"kubernetes.io/projected/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-kube-api-access-mpfvh\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.809751 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.809469    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgnm\" (UniqueName: \"kubernetes.io/projected/40daba9a-1280-486d-b7f4-047d0efdab2e-kube-api-access-blgnm\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:52.809751 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.809707    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ns6\" (UniqueName: \"kubernetes.io/projected/9713304b-d13e-4e2c-b460-6a02c9adf5b4-kube-api-access-f9ns6\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.809751 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.809718    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8729\" (UniqueName: \"kubernetes.io/projected/0977c82f-e896-4817-b4bf-eb1b056060c6-kube-api-access-d8729\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.809751 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.809724    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rvz\" (UniqueName: \"kubernetes.io/projected/eb21c178-123e-4c5c-a0f3-e59e5aab8927-kube-api-access-65rvz\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.810358 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.810335    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc45c\" (UniqueName: \"kubernetes.io/projected/a4fa7bf3-f448-451d-b46e-a885de36da57-kube-api-access-xc45c\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.818200 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.818159    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal" event={"ID":"61a071342255b3898148bdccf890b4bc","Type":"ContainerStarted","Data":"096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b"}
Nov 03 14:10:52.819887 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.819862    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" event={"ID":"86a96a710dae74d27a1c0a4680122307","Type":"ContainerStarted","Data":"dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c"}
Nov 03 14:10:52.890514 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.890463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:10:52.900694 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.900672    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:10:52.901477 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.901454    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb511bd_1e7a_44c1_b027_964d3fb57d08.slice/crio-85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7 WatchSource:0}: Error finding container 85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7: Status 404 returned error can't find the container with id 85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7
Nov 03 14:10:52.911374 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.911349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:52.911899 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.911872    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c76fe7_8625_4a41_a039_af53c4978a94.slice/crio-a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0 WatchSource:0}: Error finding container a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0: Status 404 returned error can't find the container with id a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0
Nov 03 14:10:52.922514 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.922483    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e29be6_c48c_4bc8_a601_a0e337715b61.slice/crio-f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1 WatchSource:0}: Error finding container f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1: Status 404 returned error can't find the container with id f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1
Nov 03 14:10:52.938134 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.938105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:10:52.941920 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.941901    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:10:52.945904 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.945881    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:10:52.949212 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.948831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9f6lw"
Nov 03 14:10:52.953587 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.951954    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e35132_a355_4ec6_a6f6_3ef83c5c2840.slice/crio-475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817 WatchSource:0}: Error finding container 475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817: Status 404 returned error can't find the container with id 475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817
Nov 03 14:10:52.953587 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.952410    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0977c82f_e896_4817_b4bf_eb1b056060c6.slice/crio-aa27326c957c7320f7d242266964a88ec0c783a8b1e404ec3192e4999f28b69a WatchSource:0}: Error finding container aa27326c957c7320f7d242266964a88ec0c783a8b1e404ec3192e4999f28b69a: Status 404 returned error can't find the container with id aa27326c957c7320f7d242266964a88ec0c783a8b1e404ec3192e4999f28b69a
Nov 03 14:10:52.958227 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.958200    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fa7bf3_f448_451d_b46e_a885de36da57.slice/crio-3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15 WatchSource:0}: Error finding container 3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15: Status 404 returned error can't find the container with id 3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15
Nov 03 14:10:52.960904 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.960872    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9713304b_d13e_4e2c_b460_6a02c9adf5b4.slice/crio-b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e WatchSource:0}: Error finding container b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e: Status 404 returned error can't find the container with id b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e
Nov 03 14:10:52.967547 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.967333    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:10:52.974022 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:52.973879    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:10:52.979791 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.979747    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edbb9a8_e48f_428f_9967_61dd8333133e.slice/crio-b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661 WatchSource:0}: Error finding container b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661: Status 404 returned error can't find the container with id b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661
Nov 03 14:10:52.985158 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:10:52.985132    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb21c178_123e_4c5c_a0f3_e59e5aab8927.slice/crio-96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f WatchSource:0}: Error finding container 96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f: Status 404 returned error can't find the container with id 96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f
Nov 03 14:10:53.043268 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.043185    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:10:53.286937 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.286902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:53.287041 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.286978    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:53.287095 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.287076    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:53.287143 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.287130    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:53.287195 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.287153    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.287130383 +0000 UTC m=+4.116375304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:53.287195 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.287182    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.28716801 +0000 UTC m=+4.116412920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:53.318504 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.318440    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:10:53.388375 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.388330    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:53.388546 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.388505    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:10:53.388546 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.388529    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:10:53.388546 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.388542    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:53.388702 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.388608    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:54.388587054 +0000 UTC m=+4.217831973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:53.649630 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.649532    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:52 +0000 UTC" deadline="2028-06-05 18:00:47.614618799 +0000 UTC"
Nov 03 14:10:53.649630 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.649582    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13923h49m53.965041987s"
Nov 03 14:10:53.813977 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.813943    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:53.814159 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:53.814088    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:10:53.825204 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.825167    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7t76q" event={"ID":"d6e29be6-c48c-4bc8-a601-a0e337715b61","Type":"ContainerStarted","Data":"f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1"}
Nov 03 14:10:53.827402 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.827369    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9r7l" event={"ID":"b1c76fe7-8625-4a41-a039-af53c4978a94","Type":"ContainerStarted","Data":"a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0"}
Nov 03 14:10:53.830016 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.829984    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn" event={"ID":"a4fa7bf3-f448-451d-b46e-a885de36da57","Type":"ContainerStarted","Data":"3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15"}
Nov 03 14:10:53.841616 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.841589    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerStarted","Data":"475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817"}
Nov 03 14:10:53.850228 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.850201    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kmcrp" event={"ID":"1eb511bd-1e7a-44c1-b027-964d3fb57d08","Type":"ContainerStarted","Data":"85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7"}
Nov 03 14:10:53.854079 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.854053    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f"}
Nov 03 14:10:53.881326 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.881298    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-868d4" event={"ID":"8edbb9a8-e48f-428f-9967-61dd8333133e","Type":"ContainerStarted","Data":"b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661"}
Nov 03 14:10:53.894234 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.893802    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9f6lw" event={"ID":"9713304b-d13e-4e2c-b460-6a02c9adf5b4","Type":"ContainerStarted","Data":"b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e"}
Nov 03 14:10:53.899718 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:53.899659    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k8qn5" event={"ID":"0977c82f-e896-4817-b4bf-eb1b056060c6","Type":"ContainerStarted","Data":"aa27326c957c7320f7d242266964a88ec0c783a8b1e404ec3192e4999f28b69a"}
Nov 03 14:10:54.297300 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.296662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:54.297300 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.296961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:54.297300 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.297087    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:54.297300 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.297143    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:10:56.297124363 +0000 UTC m=+6.126369284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:54.301317 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.298717    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:54.301317 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.298800    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:56.298779011 +0000 UTC m=+6.128023930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:54.398061 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.398024    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:54.398245 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.398207    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:10:54.398245 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.398226    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:10:54.398245 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.398238    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:54.398416 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.398306    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:10:56.398288323 +0000 UTC m=+6.227533242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:54.650747 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.650629    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:52 +0000 UTC" deadline="2028-04-16 23:53:09.43146277 +0000 UTC"
Nov 03 14:10:54.650747 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.650686    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12729h42m14.780779802s"
Nov 03 14:10:54.813782 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.813754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:54.813975 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.813865    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:10:54.814202 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.814173    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:54.814316 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:54.814296    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:10:54.912023 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.911990    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal" event={"ID":"61a071342255b3898148bdccf890b4bc","Type":"ContainerStarted","Data":"43829e187da0f2adee276632f1d8d6eccfbef56d624cd04bc9dbd3459f8f0a5f"}
Nov 03 14:10:54.927294 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.927265    2583 generic.go:356] "Generic (PLEG): container finished" podID="86a96a710dae74d27a1c0a4680122307" containerID="f1755eccd356e1f2576c8b7b6e8d71487dffb5b8ba9d730ad3ccb4001cfe21ab" exitCode=0
Nov 03 14:10:54.927435 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:54.927317    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" event={"ID":"86a96a710dae74d27a1c0a4680122307","Type":"ContainerDied","Data":"f1755eccd356e1f2576c8b7b6e8d71487dffb5b8ba9d730ad3ccb4001cfe21ab"}
Nov 03 14:10:55.009144 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:55.009065    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal" podStartSLOduration=4.009046106 podStartE2EDuration="4.009046106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:54.957937307 +0000 UTC m=+4.787182240" watchObservedRunningTime="2026-11-03 14:10:55.009046106 +0000 UTC m=+4.838291033"
Nov 03 14:10:55.813846 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:55.813363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:55.813846 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:55.813495    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:10:55.940776 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:55.940748    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" event={"ID":"86a96a710dae74d27a1c0a4680122307","Type":"ContainerStarted","Data":"7124ae02151f24d10bf01a2423352d709846902cc6f189a8c85efed629fa9207"}
Nov 03 14:10:56.314126 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.314048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:56.314126 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.314103    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:56.314354 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.314208    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:56.314354 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.314275    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:00.314256913 +0000 UTC m=+10.143501820 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:10:56.314691 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.314671    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:56.314766 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.314727    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:11:00.314713299 +0000 UTC m=+10.143958208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:10:56.414510 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.414475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:56.414682 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.414627    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:10:56.414682 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.414640    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:10:56.414682 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.414652    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:56.414856 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.414691    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:00.414678858 +0000 UTC m=+10.243923762 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:10:56.815270 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.812868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:56.815270 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.812970    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:10:56.815270 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.813314    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:56.815270 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:56.813407    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:10:56.945866 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.945803    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7t76q" event={"ID":"d6e29be6-c48c-4bc8-a601-a0e337715b61","Type":"ContainerStarted","Data":"1c366c1c2f6091400e398e627530f1b43cd53cfe59c81ef6690e323564a0fe54"}
Nov 03 14:10:56.952578 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.952548    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="aee30b0ad07a679f8e797233c30758d9f92b710817f4b95d5e40e60654e59fbb" exitCode=0
Nov 03 14:10:56.952717 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.952631    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerDied","Data":"aee30b0ad07a679f8e797233c30758d9f92b710817f4b95d5e40e60654e59fbb"}
Nov 03 14:10:56.991316 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.991239    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" podStartSLOduration=5.991212355 podStartE2EDuration="5.991212355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:55.982959984 +0000 UTC m=+5.812204909" watchObservedRunningTime="2026-11-03 14:10:56.991212355 +0000 UTC m=+6.820457281"
Nov 03 14:10:56.991591 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:56.991555    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7t76q" podStartSLOduration=2.273152523 podStartE2EDuration="5.991542045s" totalImagesPullingTime="3.718389522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:56.991372133 +0000 UTC m=+6.820617059" watchObservedRunningTime="2026-11-03 14:10:56.991542045 +0000 UTC m=+6.820786972"
Nov 03 14:10:57.813471 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:57.813438    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:57.813633 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:57.813540    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:10:58.172937 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.172893    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:58.173823 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.173601    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:10:58.480137 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.480103    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:10:58.702404 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.702337    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:10:58.813851 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.813821    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:10:58.813968 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:58.813948    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:10:58.814047 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.814033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:10:58.814142 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:58.814126    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:10:58.964874 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:58.964800    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-868d4" event={"ID":"8edbb9a8-e48f-428f-9967-61dd8333133e","Type":"ContainerStarted","Data":"83d6bc5ee79a843c7212f9ec77333025e5816f0070a3a08eab3617a03cee9e65"}
Nov 03 14:10:59.813672 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:59.813640    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:10:59.814104 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:10:59.813771    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:10:59.970166 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:59.969737    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn" event={"ID":"a4fa7bf3-f448-451d-b46e-a885de36da57","Type":"ContainerStarted","Data":"95b9571557af176e6c5d19fa56a6e9b5d66c08b3ab0a2edd89765537dab81eb5"}
Nov 03 14:10:59.970166 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:10:59.969824    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:11:00.347458 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.346635    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:00.347458 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.346700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:00.347458 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.346854    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:11:00.347458 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.346911    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.346894152 +0000 UTC m=+18.176139069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:11:00.347458 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.347318    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:11:00.347458 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.347365    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.347351137 +0000 UTC m=+18.176596055 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:11:00.447631 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.447602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:00.447741 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.447729    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:11:00.447798 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.447756    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:11:00.447798 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.447770    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:11:00.447896 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.447820    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:08.44780265 +0000 UTC m=+18.277047564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:11:00.607702 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.607648    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 14:11:00.650384 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.650295    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T14:11:00.607669838Z","UUID":"665e0fe4-9472-4eff-aa64-1745f833122a","Handler":null,"Name":"","Endpoint":""}
Nov 03 14:11:00.652552 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.652306    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 14:11:00.652552 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.652334    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 14:11:00.791122 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.790895    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:00.818969 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.818939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:00.819396 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.818979    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:00.819396 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.819086    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:00.819396 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:00.819181    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:00.974153 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.974123    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9f6lw" event={"ID":"9713304b-d13e-4e2c-b460-6a02c9adf5b4","Type":"ContainerStarted","Data":"7ec1353f56d76b8d6c7cd27335abe4a1e71c6f0a1e77f6304d2092d2f8631ec7"}
Nov 03 14:11:00.977030 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:00.977004    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn" event={"ID":"a4fa7bf3-f448-451d-b46e-a885de36da57","Type":"ContainerStarted","Data":"1798eb7ef1148b95248ce07364d414dddd3c9220798731574282b1129adbaf3d"}
Nov 03 14:11:01.015854 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.015822    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:01.021172 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.021132    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-868d4" podStartSLOduration=5.144725717 podStartE2EDuration="10.021121155s" totalImagesPullingTime="4.876395438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:59.008402519 +0000 UTC m=+8.837647444" watchObservedRunningTime="2026-11-03 14:11:01.021121155 +0000 UTC m=+10.850366140"
Nov 03 14:11:01.021817 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.021779    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9f6lw" podStartSLOduration=2.40498197 podStartE2EDuration="10.021770656s" totalImagesPullingTime="7.616788686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:01.020663375 +0000 UTC m=+10.849908301" watchObservedRunningTime="2026-11-03 14:11:01.021770656 +0000 UTC m=+10.851015582"
Nov 03 14:11:01.249126 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.249091    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:01.813065 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.813034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:01.813239 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:01.813159    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:01.981031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.980744    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g9r7l" event={"ID":"b1c76fe7-8625-4a41-a039-af53c4978a94","Type":"ContainerStarted","Data":"fe0a4c3fad225d2dfc38ed5996cbba59bcd759ec6ef4d704a76a68cc2bdce313"}
Nov 03 14:11:01.983144 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.983117    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn" event={"ID":"a4fa7bf3-f448-451d-b46e-a885de36da57","Type":"ContainerStarted","Data":"9454318c5ad122b5ffcf2e5416cadcae3b89673543df595c945ca85ac19d0ba7"}
Nov 03 14:11:01.985136 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.985106    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="4b1b46811728d90140eba954d27b459e4f7965c4146f7c4c6ef8194129252d83" exitCode=0
Nov 03 14:11:01.985274 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:01.985224    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerDied","Data":"4b1b46811728d90140eba954d27b459e4f7965c4146f7c4c6ef8194129252d83"}
Nov 03 14:11:02.023289 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.023222    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-g9r7l" podStartSLOduration=4.098950685 podStartE2EDuration="12.023205469s" totalImagesPullingTime="7.924254784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:02.02300736 +0000 UTC m=+11.852252286" watchObservedRunningTime="2026-11-03 14:11:02.023205469 +0000 UTC m=+11.852450397"
Nov 03 14:11:02.091009 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.090973    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:02.119368 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.119316    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn" podStartSLOduration=2.845392201 podStartE2EDuration="11.119298282s" totalImagesPullingTime="8.273906081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:02.118580453 +0000 UTC m=+11.947825380" watchObservedRunningTime="2026-11-03 14:11:02.119298282 +0000 UTC m=+11.948543209"
Nov 03 14:11:02.306455 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.306380    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:02.813250 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.813164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:02.813250 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.813223    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:02.813455 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:02.813390    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:02.813849 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:02.813816    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:02.988961 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.988925    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k8qn5" event={"ID":"0977c82f-e896-4817-b4bf-eb1b056060c6","Type":"ContainerStarted","Data":"aee7da41c84d869e1e9f327a2f4fba854f5465d00a9f84a26d2ad02c8cdead90"}
Nov 03 14:11:02.991457 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.991427    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="322564f5adbc888b1962a0f5a87ba85f4254e3b60a7a10189a9cb3de0c2316f6" exitCode=0
Nov 03 14:11:02.991584 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.991480    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerDied","Data":"322564f5adbc888b1962a0f5a87ba85f4254e3b60a7a10189a9cb3de0c2316f6"}
Nov 03 14:11:02.993241 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:02.993201    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kmcrp" event={"ID":"1eb511bd-1e7a-44c1-b027-964d3fb57d08","Type":"ContainerStarted","Data":"9d3b6d3e5191b380e7ff463a0870fa526594f3c9672882df83b38e45215a9f26"}
Nov 03 14:11:03.015310 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:03.015283    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:03.035698 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:03.035656    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k8qn5" podStartSLOduration=2.382814752 podStartE2EDuration="12.035642597s" totalImagesPullingTime="9.652827845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:03.035331502 +0000 UTC m=+12.864576429" watchObservedRunningTime="2026-11-03 14:11:03.035642597 +0000 UTC m=+12.864887524"
Nov 03 14:11:03.079260 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:03.079167    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kmcrp" podStartSLOduration=5.29792384 podStartE2EDuration="13.079155436s" totalImagesPullingTime="7.781231596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:03.078672083 +0000 UTC m=+12.907917008" watchObservedRunningTime="2026-11-03 14:11:03.079155436 +0000 UTC m=+12.908400370"
Nov 03 14:11:03.255637 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:03.255602    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:03.805518 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:03.805491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:03.813463 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:03.813444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:03.813550 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:03.813536    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:04.503914 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:04.503681    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:04.721032 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:04.721001    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:04.813576 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:04.813472    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:04.813576 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:04.813511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:04.813778 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:04.813582    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:04.813778 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:04.813714    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:04.955148 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:04.955116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:05.000289 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:05.000252    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="ef071166d03d533d40faeae6e780bb87746d8bfaed6585a9330744f305ffa324" exitCode=0
Nov 03 14:11:05.000459 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:05.000301    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerDied","Data":"ef071166d03d533d40faeae6e780bb87746d8bfaed6585a9330744f305ffa324"}
Nov 03 14:11:05.053888 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:05.053864    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9r7l_b1c76fe7-8625-4a41-a039-af53c4978a94/dns-node-resolver/0.log"
Nov 03 14:11:05.172071 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:05.172040    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:05.813888 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:05.813856    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:05.814357 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:05.813990    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:06.051420 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:06.051392    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-868d4_8edbb9a8-e48f-428f-9967-61dd8333133e/node-ca/0.log"
Nov 03 14:11:06.157156 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:06.157131    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:06.562709 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:06.562632    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:06.785438 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:06.785405    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:06.813924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:06.813693    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:06.813924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:06.813709    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:06.813924 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:06.813824    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:06.814472 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:06.813936    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:07.011166 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:07.011118    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:07.659484 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:07.659448    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:07.812894 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:07.812866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:07.813044 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:07.812986    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:08.008270 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.008244    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="435a9af9665e767c2b8856c60e02eef6ccd674b6a77379823d202339656558c7" exitCode=0
Nov 03 14:11:08.008544 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.008303    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerDied","Data":"435a9af9665e767c2b8856c60e02eef6ccd674b6a77379823d202339656558c7"}
Nov 03 14:11:08.164686 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.164548    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:08.386512 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.386482    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:08.406158 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.406134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:08.406317 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.406171    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:08.406317 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.406297    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:11:08.406317 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.406298    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:11:08.406481 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.406343    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:24.406330942 +0000 UTC m=+34.235575859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:11:08.406481 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.406356    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:11:24.406349994 +0000 UTC m=+34.235594896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:11:08.506658 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.506582    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:08.506797 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.506711    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:11:08.506797 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.506732    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:11:08.506797 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.506743    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:11:08.506797 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.506796    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:24.506780832 +0000 UTC m=+34.336025751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:11:08.615847 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.615814    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:08.813241 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.813147    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:08.813392 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.813285    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:08.813392 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.813349    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:08.813510 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:08.813487    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:08.835561 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:08.835531    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:09.013767 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.013734    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"b0ff23af073611ab3995bc088be37e4133c1893a440f8658b1a2cad86b088464"}
Nov 03 14:11:09.014159 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.013813    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"e6bf242c01800930d7da20bb1112250aa4c2dd436fb8271c6d075ff4ed79836f"}
Nov 03 14:11:09.014159 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.013870    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"956190c17b21ccb1751f9cabdf67ec854232ad155928d52f3ea48d8312aa8d49"}
Nov 03 14:11:09.014159 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.013899    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"5d52e8e1b4e07cc7c04b6d5bac35c08a096596958cb8bedf5a4b712a3922f762"}
Nov 03 14:11:09.014159 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.013932    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"ec37c6740a9a3b666a90e024a1e6787c9905bdb0d384c9a60ab54de5856bbd48"}
Nov 03 14:11:09.014159 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.013975    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"654a6e930064fffb1a43a8a927df497ae110774a2944a7536a254fe9725e8131"}
Nov 03 14:11:09.016706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.016681    2583 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="85d2fccf3b432d777b7d308140a31e3101b6c3845cdd1820c996e346488e865b" exitCode=0
Nov 03 14:11:09.016793 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.016711    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerDied","Data":"85d2fccf3b432d777b7d308140a31e3101b6c3845cdd1820c996e346488e865b"}
Nov 03 14:11:09.060635 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.060615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:09.813151 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:09.813123    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:09.813309 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:09.813221    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:10.022030 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.021986    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" event={"ID":"d6e35132-a355-4ec6-a6f6-3ef83c5c2840","Type":"ContainerStarted","Data":"9329e52535bc79ab506bf9be14447c42abab28b0f1a9854eda0fb7b79d09be8e"}
Nov 03 14:11:10.084020 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.083919    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n6kkm" podStartSLOduration=8.67510211 podStartE2EDuration="19.083904766s" totalImagesPullingTime="10.408802656s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:10.083010787 +0000 UTC m=+19.912255713" watchObservedRunningTime="2026-11-03 14:11:10.083904766 +0000 UTC m=+19.913149691"
Nov 03 14:11:10.165389 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.165358    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:10.393249 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.393207    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:10.615862 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.615782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:10.814540 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.814508    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:10.814676 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:10.814633    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:10.814720 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.814689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:10.814792 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:10.814775    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:10.834645 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:10.834622    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:11.027089 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:11.027054    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"1a7399e16dbed6ffafc5720552a3437db291586711c66aeff5d0a6cfe3e1485a"}
Nov 03 14:11:11.061418 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:11.061388    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:11.813732 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:11.813706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:11.813875 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:11.813801    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:12.155112 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:12.155034    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:12.393673 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:12.393641    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:12.618241 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:12.618197    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:12.813042 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:12.812952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:12.813176 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:12.813060    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:12.813176 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:12.813142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:12.813288 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:12.813267    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:12.871708 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:12.871682    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:13.034552 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.034506    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" event={"ID":"eb21c178-123e-4c5c-a0f3-e59e5aab8927","Type":"ContainerStarted","Data":"32bab8816a662a8a12541141af9023d19e14429306282a133aa8658acbd87a87"}
Nov 03 14:11:13.034750 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.034733    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:11:13.049748 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.049727    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:11:13.095525 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.095450    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:13.124815 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.124776    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" podStartSLOduration=7.127508613 podStartE2EDuration="22.12476526s" totalImagesPullingTime="14.997256647s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:13.124301817 +0000 UTC m=+22.953546742" watchObservedRunningTime="2026-11-03 14:11:13.12476526 +0000 UTC m=+22.954010222"
Nov 03 14:11:13.745090 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.745057    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:13.813240 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.813200    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:13.813379 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:13.813315    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:13.969071 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:13.969043    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:14.036667 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.036602    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:11:14.037019 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.037000    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:11:14.051716 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.051698    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:11:14.210475 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.210446    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:14.428643 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.428614    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:14.651718 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.651684    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:14.813045 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.812971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:14.813409 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:14.813079    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:14.813409 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:14.813130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:14.813409 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:14.813217    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:15.038895 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:15.038871    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:11:15.156607 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:15.156538    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:15.382748 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:15.382716    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:15.597954 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:15.597926    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:15.813887 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:15.813851    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:15.814336 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:15.813972    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:15.828859 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:15.827660    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:16.040854 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:16.040835    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:11:16.813596 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:16.813567    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:16.813739 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:16.813693    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:16.813795 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:16.813751    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:16.813876 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:16.813855    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:17.813587 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:17.813554    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:17.814004 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:17.813650    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:18.580700 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:18.580674    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:11:18.580916 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:18.580848    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:11:18.589730 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:18.589696    2583 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" podUID="eb21c178-123e-4c5c-a0f3-e59e5aab8927" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 14:11:18.597733 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:18.597711    2583 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl" podUID="eb21c178-123e-4c5c-a0f3-e59e5aab8927" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 14:11:18.813918 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:18.813893    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:18.814331 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:18.813982    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:18.814331 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:18.814075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:18.814331 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:18.814191    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:19.813743 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:19.813720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:19.813895 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:19.813809    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:20.814223 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:20.814194    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:20.814596 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:20.814292    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:20.814596 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:20.814341    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:20.814596 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:20.814393    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:21.812952 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:21.812921    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:21.813124 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:21.813043    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:22.556259 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:22.556213    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:11:22.556604 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:22.556350    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:11:22.556735 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:22.556709    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:11:22.813366 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:22.813298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:22.813527 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:22.813390    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:22.813527 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:22.813478    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:22.813616 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:22.813597    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:23.435797 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:23.435762    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:23.655693 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:23.655656    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:23.814012 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:23.813936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:23.814169 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:23.814064    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:23.877749 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:23.877722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:24.416024 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.415993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:24.416202 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.416036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:24.416202 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.416148    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 14:11:24.416202 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.416158    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:11:24.416202 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.416199    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret podName:9927b257-7027-4ef3-aa19-bde73b4d93b9 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:56.416187097 +0000 UTC m=+66.245432001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret") pod "global-pull-secret-syncer-8flw8" (UID: "9927b257-7027-4ef3-aa19-bde73b4d93b9") : object "kube-system"/"original-pull-secret" not registered
Nov 03 14:11:24.416375 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.416225    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs podName:40daba9a-1280-486d-b7f4-047d0efdab2e nodeName:}" failed. No retries permitted until 2026-11-03 14:11:56.416206297 +0000 UTC m=+66.245451202 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs") pod "network-metrics-daemon-9vstd" (UID: "40daba9a-1280-486d-b7f4-047d0efdab2e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 14:11:24.517093 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.517010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:24.517215 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.517172    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 14:11:24.517215 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.517195    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 14:11:24.517215 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.517208    2583 projected.go:196] Error preparing data for projected volume kube-api-access-7zlms for pod openshift-network-diagnostics/network-check-target-s7z9p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:11:24.517390 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.517289    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms podName:167d067e-3824-4add-b57b-9ba2bfc6c200 nodeName:}" failed. No retries permitted until 2026-11-03 14:11:56.517270036 +0000 UTC m=+66.346514942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zlms" (UniqueName: "kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms") pod "network-check-target-s7z9p" (UID: "167d067e-3824-4add-b57b-9ba2bfc6c200") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 14:11:24.533758 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.533733    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8flw8"]
Nov 03 14:11:24.533900 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.533845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:24.533962 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.533932    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:24.536910 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.536886    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s7z9p"]
Nov 03 14:11:24.537046 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.537005    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:24.537147 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.537130    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:24.537634 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.537614    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vstd"]
Nov 03 14:11:24.537736 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:24.537723    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:24.537840 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:24.537814    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:25.314977 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:25.314948    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9r7l_b1c76fe7-8625-4a41-a039-af53c4978a94/dns-node-resolver/0.log"
Nov 03 14:11:25.812993 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:25.812929    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:25.812993 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:25.812951    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:25.813159 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:25.813035    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:25.813159 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:25.813121    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:26.325891 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:26.325863    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:26.512196 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:26.512168    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-868d4_8edbb9a8-e48f-428f-9967-61dd8333133e/node-ca/0.log"
Nov 03 14:11:26.559285 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:26.559254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:26.813762 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:26.813598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:26.813909 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:26.813839    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:27.578568 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:27.578536    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:27.800857 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:27.800827    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:27.813727 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:27.813706    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:27.813871 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:27.813848    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9vstd" podUID="40daba9a-1280-486d-b7f4-047d0efdab2e"
Nov 03 14:11:27.813919 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:27.813878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:27.813962 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:27.813947    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8flw8" podUID="9927b257-7027-4ef3-aa19-bde73b4d93b9"
Nov 03 14:11:28.813772 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:28.813740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:28.814141 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:28.813871    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s7z9p" podUID="167d067e-3824-4add-b57b-9ba2bfc6c200"
Nov 03 14:11:29.019039 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.019011    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:29.424774 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.424751    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeReady"
Nov 03 14:11:29.424907 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.424856    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 14:11:29.493492 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.493467    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fwrl7"]
Nov 03 14:11:29.515205 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.515179    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pkm2z"]
Nov 03 14:11:29.515334 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.515320    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.531218 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.531195    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fwrl7"]
Nov 03 14:11:29.531345 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.531252    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkm2z"]
Nov 03 14:11:29.531345 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.531279    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kws78"]
Nov 03 14:11:29.531345 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.531309    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.532615 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.532601    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 14:11:29.532702 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.532687    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghsgd\""
Nov 03 14:11:29.532782 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.532764    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 14:11:29.545386 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.545372    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.548356 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.548336    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 14:11:29.549094 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.549077    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kws78"]
Nov 03 14:11:29.633309 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633288    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 14:11:29.633410 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633358    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 14:11:29.633507 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633493    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 14:11:29.633538 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633514    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 14:11:29.633538 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633525    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 14:11:29.633598 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633565    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlrhs\""
Nov 03 14:11:29.633598 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633570    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 14:11:29.633598 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.633574    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r6xhc\""
Nov 03 14:11:29.655989 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.655962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2d69636-c98b-4456-93be-168d230451b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.655989 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.655990    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3196f084-b0f3-4145-be7f-0d2b63dd8019-tmp-dir\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.656155 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656008    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2d69636-c98b-4456-93be-168d230451b6-crio-socket\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.656155 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656022    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2d69636-c98b-4456-93be-168d230451b6-data-volume\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.656155 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656125    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3196f084-b0f3-4145-be7f-0d2b63dd8019-metrics-tls\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.656293 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3196f084-b0f3-4145-be7f-0d2b63dd8019-config-volume\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.656293 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656223    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d998c6-eb92-4c4e-b35e-fa3703de95ed-cert\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.656293 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656258    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjw8\" (UniqueName: \"kubernetes.io/projected/15d998c6-eb92-4c4e-b35e-fa3703de95ed-kube-api-access-tjjw8\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.656404 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656298    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxbk\" (UniqueName: \"kubernetes.io/projected/f2d69636-c98b-4456-93be-168d230451b6-kube-api-access-9xxbk\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.656404 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2d69636-c98b-4456-93be-168d230451b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.656404 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.656401    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9mw\" (UniqueName: \"kubernetes.io/projected/3196f084-b0f3-4145-be7f-0d2b63dd8019-kube-api-access-7h9mw\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757023 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.756972    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3196f084-b0f3-4145-be7f-0d2b63dd8019-config-volume\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757023 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.756994    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d998c6-eb92-4c4e-b35e-fa3703de95ed-cert\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.757023 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjw8\" (UniqueName: \"kubernetes.io/projected/15d998c6-eb92-4c4e-b35e-fa3703de95ed-kube-api-access-tjjw8\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757027    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxbk\" (UniqueName: \"kubernetes.io/projected/f2d69636-c98b-4456-93be-168d230451b6-kube-api-access-9xxbk\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757063    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2d69636-c98b-4456-93be-168d230451b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757096    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9mw\" (UniqueName: \"kubernetes.io/projected/3196f084-b0f3-4145-be7f-0d2b63dd8019-kube-api-access-7h9mw\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2d69636-c98b-4456-93be-168d230451b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3196f084-b0f3-4145-be7f-0d2b63dd8019-tmp-dir\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2d69636-c98b-4456-93be-168d230451b6-crio-socket\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757217 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2d69636-c98b-4456-93be-168d230451b6-data-volume\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757582 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757263    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3196f084-b0f3-4145-be7f-0d2b63dd8019-metrics-tls\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757582 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757516    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2d69636-c98b-4456-93be-168d230451b6-crio-socket\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757684 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757666    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3196f084-b0f3-4145-be7f-0d2b63dd8019-tmp-dir\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757742 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757718    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3196f084-b0f3-4145-be7f-0d2b63dd8019-config-volume\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.757822 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757793    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2d69636-c98b-4456-93be-168d230451b6-data-volume\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.757856 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.757831    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2d69636-c98b-4456-93be-168d230451b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.760872 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.760852    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3196f084-b0f3-4145-be7f-0d2b63dd8019-metrics-tls\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.760960 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.760871    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2d69636-c98b-4456-93be-168d230451b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.760960 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.760890    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d998c6-eb92-4c4e-b35e-fa3703de95ed-cert\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.779379 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.779354    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxbk\" (UniqueName: \"kubernetes.io/projected/f2d69636-c98b-4456-93be-168d230451b6-kube-api-access-9xxbk\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:29.779540 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.779523    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjw8\" (UniqueName: \"kubernetes.io/projected/15d998c6-eb92-4c4e-b35e-fa3703de95ed-kube-api-access-tjjw8\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.779600 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.779582    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9mw\" (UniqueName: \"kubernetes.io/projected/3196f084-b0f3-4145-be7f-0d2b63dd8019-kube-api-access-7h9mw\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.813626 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.813601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:29.813773 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.813756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:29.831342 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.831323    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\""
Nov 03 14:11:29.831697 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.831357    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\""
Nov 03 14:11:29.831697 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.831407    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 14:11:29.831697 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.831479    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 14:11:29.852413 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.852387    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:29.870542 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.870529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:11:29.910377 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:29.910351    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:11:30.203459 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.203427    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pkm2z"]
Nov 03 14:11:30.207742 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:30.207719    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d998c6_eb92_4c4e_b35e_fa3703de95ed.slice/crio-76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda WatchSource:0}: Error finding container 76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda: Status 404 returned error can't find the container with id 76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda
Nov 03 14:11:30.218512 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.218491    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fwrl7"]
Nov 03 14:11:30.221303 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:30.221280    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3196f084_b0f3_4145_be7f_0d2b63dd8019.slice/crio-e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13 WatchSource:0}: Error finding container e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13: Status 404 returned error can't find the container with id e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13
Nov 03 14:11:30.230022 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.230001    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kws78"]
Nov 03 14:11:30.232403 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:30.232381    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d69636_c98b_4456_93be_168d230451b6.slice/crio-7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af WatchSource:0}: Error finding container 7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af: Status 404 returned error can't find the container with id 7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af
Nov 03 14:11:30.809742 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.809702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:30.817880 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.817860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:30.835577 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.835552    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 14:11:30.836162 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.835678    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 14:11:30.836162 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:30.835798    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\""
Nov 03 14:11:31.077896 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.077816    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kws78" event={"ID":"f2d69636-c98b-4456-93be-168d230451b6","Type":"ContainerStarted","Data":"e8ef15540841e17c799d004b769eedaab572bb3b689e5d7c26477e9bcf4aabf5"}
Nov 03 14:11:31.078022 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.077923    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kws78" event={"ID":"f2d69636-c98b-4456-93be-168d230451b6","Type":"ContainerStarted","Data":"7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af"}
Nov 03 14:11:31.079699 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.079671    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkm2z" event={"ID":"15d998c6-eb92-4c4e-b35e-fa3703de95ed","Type":"ContainerStarted","Data":"76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda"}
Nov 03 14:11:31.081532 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.081501    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fwrl7" event={"ID":"3196f084-b0f3-4145-be7f-0d2b63dd8019","Type":"ContainerStarted","Data":"e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13"}
Nov 03 14:11:31.101340 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.101312    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n87hh"]
Nov 03 14:11:31.103365 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.103308    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:31.114461 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.114441    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.144662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.144636    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 14:11:31.144822 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.144708    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjlcb\""
Nov 03 14:11:31.144882 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.144866    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 14:11:31.157668 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.157646    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 14:11:31.157668 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.157657    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 14:11:31.158121 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.157654    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 14:11:31.158121 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.157989    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 14:11:31.269623 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269586    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-accelerators-collector-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.269769 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269666    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.269823 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-tls\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.269823 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269806    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-wtmp\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.269917 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-metrics-client-ca\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.269917 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269880    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-sys\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.270015 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.269971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-textfile\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.270068 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.270050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-root\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.270118 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.270082    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7vl\" (UniqueName: \"kubernetes.io/projected/e3ccb6ab-b665-4753-96a2-9ce1fa145666-kube-api-access-wh7vl\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.335133 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.335057    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:31.371369 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-accelerators-collector-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371541 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371541 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371463    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-tls\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371541 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-wtmp\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371541 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371515    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-metrics-client-ca\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371541 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-sys\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371807 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371569    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-textfile\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371807 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-root\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.371807 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371631    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7vl\" (UniqueName: \"kubernetes.io/projected/e3ccb6ab-b665-4753-96a2-9ce1fa145666-kube-api-access-wh7vl\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.372441 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.371978    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-accelerators-collector-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.372441 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.372072    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-sys\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.372441 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.372113    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-root\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.372441 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.372159    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-wtmp\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.372683 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.372460    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-textfile\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.372683 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.372478    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-metrics-client-ca\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.375779 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.375755    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.375875 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.375818    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-tls\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.406454 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.406404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7vl\" (UniqueName: \"kubernetes.io/projected/e3ccb6ab-b665-4753-96a2-9ce1fa145666-kube-api-access-wh7vl\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.485096 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.485071    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:11:31.516009 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:31.515981    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ccb6ab_b665_4753_96a2_9ce1fa145666.slice/crio-cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c WatchSource:0}: Error finding container cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c: Status 404 returned error can't find the container with id cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c
Nov 03 14:11:31.634946 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.634868    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:31.856374 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:31.856338    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:32.088738 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.088628    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kws78" event={"ID":"f2d69636-c98b-4456-93be-168d230451b6","Type":"ContainerStarted","Data":"147ed96720c0103a149a157fdbcda2ad92b59f568252cd04e9d005fbc20bd02f"}
Nov 03 14:11:32.090143 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.090070    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n87hh" event={"ID":"e3ccb6ab-b665-4753-96a2-9ce1fa145666","Type":"ContainerStarted","Data":"cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c"}
Nov 03 14:11:32.096534 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.096489    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:32.325857 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.325820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:32.477761 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.477730    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:11:32.506486 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.506463    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:11:32.506609 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.506592    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.524600 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524538    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 14:11:32.524706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524639    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 14:11:32.524706 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524693    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 14:11:32.524815 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524719    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ndkr\""
Nov 03 14:11:32.524815 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524727    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 14:11:32.524815 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524788    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 14:11:32.524815 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524789    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 14:11:32.524964 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524831    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 14:11:32.524964 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.524847    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 14:11:32.538841 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.538814    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 14:11:32.545626 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.545603    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581594    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581647    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581702    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581720    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581735    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581751    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581822    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.582095 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.581900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682490 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682490 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682604    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682629    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682662 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682986 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682986 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682766    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682986 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.682986 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.682811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.683178 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.683143    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.684779 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.683404    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.684779 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:32.684118    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle podName:fcf9acbd-ada8-42c9-a659-32e906773a1f nodeName:}" failed. No retries permitted until 2026-11-03 14:11:33.184098283 +0000 UTC m=+43.013343203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f") : configmap references non-existent config key: ca-bundle.crt
Nov 03 14:11:32.686616 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.686589    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.686743 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.686621    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.687881 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.687861    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.693169 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.692721    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.693169 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.692877    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.693169 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.692978    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.693406 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.693365    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.693852 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.693733    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.694597 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.694556    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.708135 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.708080    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:32.766816 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:32.766790    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:33.093991 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.093916    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fwrl7" event={"ID":"3196f084-b0f3-4145-be7f-0d2b63dd8019","Type":"ContainerStarted","Data":"1b6b2f5b4be0686893b6e672b9856a2d83db0e003f14e4937fe559350eef8e5d"}
Nov 03 14:11:33.093991 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.093963    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fwrl7" event={"ID":"3196f084-b0f3-4145-be7f-0d2b63dd8019","Type":"ContainerStarted","Data":"4e08c7a98045e05d65b7aa18b30821392aa00955e1a28f815f550beec648d03c"}
Nov 03 14:11:33.094378 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.094049    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:33.138624 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.138581    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-fwrl7" podStartSLOduration=1.9332479139999998 podStartE2EDuration="4.138568436s" totalImagesPullingTime="2.205320522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:33.137717754 +0000 UTC m=+42.966962679" watchObservedRunningTime="2026-11-03 14:11:33.138568436 +0000 UTC m=+42.967813353"
Nov 03 14:11:33.185946 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.185921    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:33.186096 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:33.186080    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle podName:fcf9acbd-ada8-42c9-a659-32e906773a1f nodeName:}" failed. No retries permitted until 2026-11-03 14:11:34.186063431 +0000 UTC m=+44.015308349 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f") : configmap references non-existent config key: ca-bundle.crt
Nov 03 14:11:33.264446 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.264421    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"]
Nov 03 14:11:33.294289 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.294271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.295394 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.295372    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"]
Nov 03 14:11:33.314558 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314534    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 14:11:33.314656 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314570    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 14:11:33.314656 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314589    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 14:11:33.314656 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314570    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 14:11:33.314656 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314602    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f799n8dkgte3s\""
Nov 03 14:11:33.314656 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314596    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 14:11:33.314656 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.314647    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l8fm6\""
Nov 03 14:11:33.388451 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388451 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-grpc-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388625 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-metrics-client-ca\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388625 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46t45\" (UniqueName: \"kubernetes.io/projected/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-kube-api-access-46t45\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388701 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-tmp\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388763 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388733    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388829 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388809    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388873 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.388930 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.388912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.452975 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.452951    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:33.489399 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-metrics-client-ca\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.489493 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t45\" (UniqueName: \"kubernetes.io/projected/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-kube-api-access-46t45\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.489493 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-tmp\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.489493 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.489602 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489496    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.489713 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.489912 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.490017 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.489966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.490017 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.490002    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-tmp\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.490145 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.490006    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-grpc-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.490145 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.490055    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-metrics-client-ca\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.492533 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.492513    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.492641 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.492620    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.492704 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.492673    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.492977 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.492956    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-grpc-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.493050 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.493030    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.493651 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.493634    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.511726 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.511682    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t45\" (UniqueName: \"kubernetes.io/projected/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-kube-api-access-46t45\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:33.667525 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.667494    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:33.707740 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:33.707718    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:34.062553 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.062378    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"]
Nov 03 14:11:34.065301 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:34.065274    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf983b9ee_1e60_4ec4_a196_e40be0d0ccd9.slice/crio-12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae WatchSource:0}: Error finding container 12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae: Status 404 returned error can't find the container with id 12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae
Nov 03 14:11:34.097759 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.097725    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pkm2z" event={"ID":"15d998c6-eb92-4c4e-b35e-fa3703de95ed","Type":"ContainerStarted","Data":"dabea507f96279b5386917797f7369c5d49a475711413fd863fa333c81a34d83"}
Nov 03 14:11:34.099801 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.099773    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kws78" event={"ID":"f2d69636-c98b-4456-93be-168d230451b6","Type":"ContainerStarted","Data":"4da5a07f37d4ae49bcd010f24f0360a8703c6cfd0cac631ca4e0a3da42af19f6"}
Nov 03 14:11:34.101041 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.101019    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae"}
Nov 03 14:11:34.103014 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.102993    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3ccb6ab-b665-4753-96a2-9ce1fa145666" containerID="78834032a7ac6445acd4135cf0ad5c500a4e96699492d4c78de8450c9e18043c" exitCode=0
Nov 03 14:11:34.103108 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.103076    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n87hh" event={"ID":"e3ccb6ab-b665-4753-96a2-9ce1fa145666","Type":"ContainerDied","Data":"78834032a7ac6445acd4135cf0ad5c500a4e96699492d4c78de8450c9e18043c"}
Nov 03 14:11:34.142881 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.142844    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pkm2z" podStartSLOduration=1.6602869519999999 podStartE2EDuration="5.14283197s" totalImagesPullingTime="3.482545018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:34.142103292 +0000 UTC m=+43.971348222" watchObservedRunningTime="2026-11-03 14:11:34.14283197 +0000 UTC m=+43.972076888"
Nov 03 14:11:34.193715 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.193672    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kws78" podStartSLOduration=1.958631977 podStartE2EDuration="5.193658948s" totalImagesPullingTime="3.235026971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 14:11:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:34.193449109 +0000 UTC m=+44.022694047" watchObservedRunningTime="2026-11-03 14:11:34.193658948 +0000 UTC m=+44.022903875"
Nov 03 14:11:34.195253 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.195206    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:34.195548 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:34.195519    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle podName:fcf9acbd-ada8-42c9-a659-32e906773a1f nodeName:}" failed. No retries permitted until 2026-11-03 14:11:36.195500753 +0000 UTC m=+46.024745675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f") : configmap references non-existent config key: ca-bundle.crt
Nov 03 14:11:34.378521 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.378448    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:34.636713 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:34.636642    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:35.109381 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.109337    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n87hh" event={"ID":"e3ccb6ab-b665-4753-96a2-9ce1fa145666","Type":"ContainerStarted","Data":"6d9f99c1851a5346a8807f925e04e81769e113919a890e057803448c7201f900"}
Nov 03 14:11:35.109840 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.109397    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n87hh" event={"ID":"e3ccb6ab-b665-4753-96a2-9ce1fa145666","Type":"ContainerStarted","Data":"281567072e6634ccee9843961c6641ddd587f51ceec3c5d0dd735e14a175c32d"}
Nov 03 14:11:35.184781 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.184738    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n87hh" podStartSLOduration=2.353217923 podStartE2EDuration="4.184724841s" totalImagesPullingTime="1.729235438s" totalInitContainerRuntime="102.27148ms" isStatefulPod=true podCreationTimestamp="2026-11-03 14:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:35.17167369 +0000 UTC m=+45.000918614" watchObservedRunningTime="2026-11-03 14:11:35.184724841 +0000 UTC m=+45.013969762"
Nov 03 14:11:35.566794 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.566766    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:35.696721 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.696686    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6679c484c8-thcsr"]
Nov 03 14:11:35.698970 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.698947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.719406 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.719376    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 14:11:35.719517 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.719435    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 14:11:35.719577 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.719520    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-21aqngq8nfakl\""
Nov 03 14:11:35.719680 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.719592    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 14:11:35.719680 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.719628    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 14:11:35.719787 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.719696    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tpn6s\""
Nov 03 14:11:35.720953 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.720933    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6679c484c8-thcsr"]
Nov 03 14:11:35.807127 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-tls\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.807127 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.807392 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-audit-log\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.807392 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807271    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-metrics-server-audit-profiles\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.807392 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-client-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.807392 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807335    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmsg\" (UniqueName: \"kubernetes.io/projected/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-kube-api-access-7kmsg\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.807596 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.807492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-client-certs\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.812063 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.812036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:35.813157 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.813134    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"]
Nov 03 14:11:35.815518 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.815490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:35.836050 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.836027    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"]
Nov 03 14:11:35.836394 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.836375    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 14:11:35.836394 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.836382    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nkgzv\""
Nov 03 14:11:35.908991 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.908953    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-audit-log\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.908991 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909005    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbssx\" (UniqueName: \"kubernetes.io/projected/9225fde2-c7af-4b4e-80c0-6a473f655fec-kube-api-access-mbssx\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:35.909278 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909041    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-metrics-server-audit-profiles\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.909278 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-client-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.909390 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909278    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmsg\" (UniqueName: \"kubernetes.io/projected/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-kube-api-access-7kmsg\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.909445 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-client-certs\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.909497 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909456    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9225fde2-c7af-4b4e-80c0-6a473f655fec-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:35.909551 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-tls\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.909551 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909528    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-audit-log\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.909660 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.909548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.910192 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.910171    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-metrics-server-audit-profiles\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.910315 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.910198    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.913573 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.913551    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-client-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.913732 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.913717    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-tls\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.913941 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.913927    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-client-certs\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:35.931892 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:35.931870    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmsg\" (UniqueName: \"kubernetes.io/projected/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-kube-api-access-7kmsg\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:36.010048 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.010017    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9225fde2-c7af-4b4e-80c0-6a473f655fec-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:36.010152 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.010075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbssx\" (UniqueName: \"kubernetes.io/projected/9225fde2-c7af-4b4e-80c0-6a473f655fec-kube-api-access-mbssx\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:36.011912 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.011889    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:36.012151 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.012133    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9225fde2-c7af-4b4e-80c0-6a473f655fec-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:36.032430 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.032406    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbssx\" (UniqueName: \"kubernetes.io/projected/9225fde2-c7af-4b4e-80c0-6a473f655fec-kube-api-access-mbssx\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:36.035794 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.035767    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:36.117267 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.117224    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"ea16630187d65b6e32447c3720448e252654c140f46126edbdcd42db2f825b55"}
Nov 03 14:11:36.126500 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.126298    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:36.211349 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.211321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:36.212426 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.212403    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:36.273630 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.273609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:11:36.335537 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.335454    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6679c484c8-thcsr"]
Nov 03 14:11:36.337400 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:36.337374    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc6d4db_06d7_4767_bef1_d90cfb51dc88.slice/crio-84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4 WatchSource:0}: Error finding container 84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4: Status 404 returned error can't find the container with id 84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4
Nov 03 14:11:36.419410 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.419388    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"]
Nov 03 14:11:36.421428 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:36.421408    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9225fde2_c7af_4b4e_80c0_6a473f655fec.slice/crio-9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf WatchSource:0}: Error finding container 9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf: Status 404 returned error can't find the container with id 9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf
Nov 03 14:11:36.583605 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.583576    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:11:36.586696 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:36.586670    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf9acbd_ada8_42c9_a659_32e906773a1f.slice/crio-457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971 WatchSource:0}: Error finding container 457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971: Status 404 returned error can't find the container with id 457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971
Nov 03 14:11:36.784871 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:36.784790    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:37.127429 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.127399    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"ffd12e069e392c0386f59ff002c062e91af824bbe6e01e334104e3a9b596b2fe"}
Nov 03 14:11:37.127780 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.127483    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"b2fc16f39cba569c4c1017e5fad0e367ccf567c0d8a5b825a31fc066fed4505d"}
Nov 03 14:11:37.127780 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.127526    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"20b6aecc042f8ae28570284fb39502c085a93a1922f265b159ec92de051d8e32"}
Nov 03 14:11:37.127780 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.127565    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"4bf36666c720c5ff320c3c4bb1283d0790845e817a74a62ba19b6bf204a5fec0"}
Nov 03 14:11:37.129889 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.129863    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr" event={"ID":"0bc6d4db-06d7-4767-bef1-d90cfb51dc88","Type":"ContainerStarted","Data":"84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4"}
Nov 03 14:11:37.132025 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.131997    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971"}
Nov 03 14:11:37.134726 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.134702    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl" event={"ID":"9225fde2-c7af-4b4e-80c0-6a473f655fec","Type":"ContainerStarted","Data":"9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf"}
Nov 03 14:11:37.955356 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.955330    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:11:37.958505 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.958477    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:37.976833 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.976808    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 14:11:37.978272 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.978255    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwv7d\""
Nov 03 14:11:37.978360 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.978271    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 14:11:37.978360 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.978222    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 14:11:37.978360 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.978299    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9g39dtaijeu5d\""
Nov 03 14:11:37.978360 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.978302    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 14:11:37.988109 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.988083    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 14:11:37.988109 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.988088    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 14:11:37.988277 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.988117    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 14:11:37.988277 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.988274    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 14:11:37.991530 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:37.991514    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 14:11:38.001389 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.001369    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 14:11:38.001471 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.001437    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 14:11:38.001745 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.001715    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 14:11:38.003609 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.003588    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 14:11:38.015067 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.015043    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 14:11:38.015990 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.015972    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:11:38.072110 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.072077    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:38.127280 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127244    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127374 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127292    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127374 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127435 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127383    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127435 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127400    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127435 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127427    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127541    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127571    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127593    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127717    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127734    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127816    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127893    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.127922 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.128307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.127967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.128307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.128046    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.140725 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.140699    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" event={"ID":"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9","Type":"ContainerStarted","Data":"deb5da7b3b783d09542a884ff3936790d69bb843a3382637e80eb37ef1a439e4"}
Nov 03 14:11:38.140861 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.140843    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:38.142279 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.142258    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr" event={"ID":"0bc6d4db-06d7-4767-bef1-d90cfb51dc88","Type":"ContainerStarted","Data":"a04075b662f57ea03f748c6a69ce042977e1f6dccf891b4bada72fa9c06de6fe"}
Nov 03 14:11:38.143760 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.143733    2583 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7" exitCode=0
Nov 03 14:11:38.143835 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.143804    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7"}
Nov 03 14:11:38.145289 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.145268    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl" event={"ID":"9225fde2-c7af-4b4e-80c0-6a473f655fec","Type":"ContainerStarted","Data":"c1dde2ec10d7f90c9d1cc22ca4b2101311762a35362c15a14e20eb27d7df2b87"}
Nov 03 14:11:38.145467 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.145453    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:38.150674 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.150657    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:11:38.191423 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.191358    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw" podStartSLOduration=2.510688114 podStartE2EDuration="5.191346261s" totalImagesPullingTime="2.680658147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:38.191175201 +0000 UTC m=+48.020420126" watchObservedRunningTime="2026-11-03 14:11:38.191346261 +0000 UTC m=+48.020591183"
Nov 03 14:11:38.228627 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.228764 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228644    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.228764 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.228764 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.228924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228773    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.228924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228820    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.228924 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.229062 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.228958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.229113 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.229065    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.229113 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.229090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.229882    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.229962    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230045    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230220    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230271    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230324    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230405    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230450    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230462    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.230509    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.231458    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.232307 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.231961    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.233083 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.232483    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.233083 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.232848    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.233083 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.232999    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.233621 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.233578    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.234623 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.234382    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.234747 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.234727    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.235031 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.235001    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.235154 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.235135    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.235295 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.235223    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.235402 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.235348    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.236024 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.236003    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.236084 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.236060    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.236126 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.236108    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.236904 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.236883    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.242111 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.242065    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr" podStartSLOduration=1.66330647 podStartE2EDuration="3.242048701s" totalImagesPullingTime="1.578742231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:38.240818321 +0000 UTC m=+48.070063247" watchObservedRunningTime="2026-11-03 14:11:38.242048701 +0000 UTC m=+48.071293627"
Nov 03 14:11:38.258406 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.258384    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.279348 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.279325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:38.293489 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.293452    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl" podStartSLOduration=1.8995842 podStartE2EDuration="3.293441219s" totalImagesPullingTime="1.393857019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:38.293147302 +0000 UTC m=+48.122392227" watchObservedRunningTime="2026-11-03 14:11:38.293441219 +0000 UTC m=+48.122686144"
Nov 03 14:11:38.602970 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:38.602947    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:11:38.604866 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:38.604826    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73da9fd2_cd6c_44b8_9bf8_abdfab16d26f.slice/crio-7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6 WatchSource:0}: Error finding container 7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6: Status 404 returned error can't find the container with id 7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6
Nov 03 14:11:39.150315 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:39.150279    2583 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4" exitCode=0
Nov 03 14:11:39.150744 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:39.150379    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"}
Nov 03 14:11:39.150744 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:39.150464    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6"}
Nov 03 14:11:40.157630 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.157587    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718"}
Nov 03 14:11:40.158116 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.157680    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed"}
Nov 03 14:11:40.158116 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.157729    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4"}
Nov 03 14:11:40.158116 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.157773    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b"}
Nov 03 14:11:40.158116 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.157831    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda"}
Nov 03 14:11:40.158116 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.157871    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerStarted","Data":"622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0"}
Nov 03 14:11:40.244311 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:40.244255    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.813433261 podStartE2EDuration="8.244222729s" totalImagesPullingTime="2.410513198s" totalInitContainerRuntime="20.27627ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:40.23072505 +0000 UTC m=+50.059969977" watchObservedRunningTime="2026-11-03 14:11:40.244222729 +0000 UTC m=+50.073467659"
Nov 03 14:11:42.162201 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:42.162170    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:42.166823 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:42.166800    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"}
Nov 03 14:11:42.380703 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:42.380680    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:42.633479 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:42.633403    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:43.105928 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.105904    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:11:43.174793 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.174755    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"}
Nov 03 14:11:43.175195 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.174837    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"}
Nov 03 14:11:43.175195 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.174876    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"}
Nov 03 14:11:43.175195 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.174914    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"}
Nov 03 14:11:43.175195 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.174951    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerStarted","Data":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"}
Nov 03 14:11:43.248136 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.248085    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.307063002 podStartE2EDuration="6.248067715s" totalImagesPullingTime="2.924217176s" totalInitContainerRuntime="16.787537ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:43.247804184 +0000 UTC m=+53.077049108" watchObservedRunningTime="2026-11-03 14:11:43.248067715 +0000 UTC m=+53.077312639"
Nov 03 14:11:43.280153 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.280119    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:11:43.343993 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.343963    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:43.560042 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.560009    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:43.948722 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:43.948694    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:44.167522 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:44.167496    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:11:44.198480 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:44.198451    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:45.869033 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:45.868999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:46.120859 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:46.120790    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:48.600298 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:48.600268    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:11:52.303994 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:52.303958    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:52.521187 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:52.521158    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:54.737196 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:54.737159    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:54.955061 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:54.955029    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:55.394786 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:55.394754    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:55.639006 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:55.638976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:56.012834 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.012800    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:56.013215 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.012862    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:11:56.491271 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.491217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:56.491446 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.491288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:56.519007 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.518984    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 14:11:56.519007 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.518984    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 14:11:56.524199 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.524173    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:56.524337 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.524304    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:56.575004 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.574979    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\""
Nov 03 14:11:56.581073 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.581053    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:11:56.587380 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.587363    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\""
Nov 03 14:11:56.592672 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.592650    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:56.592815 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.592801    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:11:56.607146 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.607117    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 14:11:56.627567 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.627541    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 14:11:56.637061 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.637003    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:56.656198 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.656173    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\""
Nov 03 14:11:56.662325 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.662308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:11:56.882915 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.882894    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9vstd"]
Nov 03 14:11:56.885217 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:56.885191    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40daba9a_1280_486d_b7f4_047d0efdab2e.slice/crio-d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6 WatchSource:0}: Error finding container d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6: Status 404 returned error can't find the container with id d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6
Nov 03 14:11:56.897246 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.897207    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8flw8"]
Nov 03 14:11:56.898753 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:56.898734    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9927b257_7027_4ef3_aa19_bde73b4d93b9.slice/crio-da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363 WatchSource:0}: Error finding container da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363: Status 404 returned error can't find the container with id da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363
Nov 03 14:11:56.957292 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:56.957266    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s7z9p"]
Nov 03 14:11:56.959525 ip-10-0-128-190 kubenswrapper[2583]: W1103 14:11:56.959503    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167d067e_3824_4add_b57b_9ba2bfc6c200.slice/crio-5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc WatchSource:0}: Error finding container 5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc: Status 404 returned error can't find the container with id 5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc
Nov 03 14:11:57.221552 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:57.221514    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s7z9p" event={"ID":"167d067e-3824-4add-b57b-9ba2bfc6c200","Type":"ContainerStarted","Data":"5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc"}
Nov 03 14:11:57.222820 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:57.222793    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8flw8" event={"ID":"9927b257-7027-4ef3-aa19-bde73b4d93b9","Type":"ContainerStarted","Data":"da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363"}
Nov 03 14:11:57.223960 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:57.223938    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vstd" event={"ID":"40daba9a-1280-486d-b7f4-047d0efdab2e","Type":"ContainerStarted","Data":"d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6"}
Nov 03 14:11:57.584481 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:57.584398    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:57.809527 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:57.809491    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:58.236112 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:58.236065    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vstd" event={"ID":"40daba9a-1280-486d-b7f4-047d0efdab2e","Type":"ContainerStarted","Data":"cad77a78f40eb105d09fe168cd3cd55561a7f9a8ab06c88bf5f1b8a42c71c772"}
Nov 03 14:11:58.236546 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:58.236171    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9vstd" event={"ID":"40daba9a-1280-486d-b7f4-047d0efdab2e","Type":"ContainerStarted","Data":"710acce7dbb9622bd0a8285c2b1f905059798f4f0371ac511c71fd53e2dc865d"}
Nov 03 14:11:58.281521 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:58.281220    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9vstd" podStartSLOduration=66.172087727 podStartE2EDuration="1m7.281203879s" totalImagesPullingTime="1.109116152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:11:58.280748631 +0000 UTC m=+68.109993548" watchObservedRunningTime="2026-11-03 14:11:58.281203879 +0000 UTC m=+68.110448806"
Nov 03 14:11:58.515109 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:58.515072    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:58.743224 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:58.743177    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:59.338476 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:59.338440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:59.566348 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:11:59.566310    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:11:59.946431 ip-10-0-128-190 kubenswrapper[2583]: E1103 14:11:59.946395    2583 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:12:01.254343 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:12:01.254307    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s7z9p" event={"ID":"167d067e-3824-4add-b57b-9ba2bfc6c200","Type":"ContainerStarted","Data":"6fe39b97d4851cd736ffb5015a6cfa6e7c66b66a60d65323a9a94de19c994183"}
Nov 03 14:12:01.254784 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:12:01.254557    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:12:01.299439 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:12:01.299393    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s7z9p" podStartSLOduration=66.935326632 podStartE2EDuration="1m10.299380271s" totalImagesPullingTime="3.364053639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:01.298510849 +0000 UTC m=+71.127755774" watchObservedRunningTime="2026-11-03 14:12:01.299380271 +0000 UTC m=+71.128625199"
Nov 03 14:12:01.893351 ip-10-0-128-190 kubenswrapper[2583]: I1103 14:12:01.893208    2583 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 14:12:01.893402 ip-10-0-128-190 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 14:12:01.904591 ip-10-0-128-190 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 14:12:01.904757 ip-10-0-128-190 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 14:12:01.904927 ip-10-0-128-190 systemd[1]: kubelet.service: Consumed 4.408s CPU time.
Nov 03 14:12:01.919692 ip-10-0-128-190 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 14:12:01.979082 ip-10-0-128-190 kubenswrapper[6481]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 14:12:01.979082 ip-10-0-128-190 kubenswrapper[6481]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 14:12:01.979082 ip-10-0-128-190 kubenswrapper[6481]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 14:12:01.979082 ip-10-0-128-190 kubenswrapper[6481]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 14:12:01.981125 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981039    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:01.981125 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981126    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981130    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981134    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981137    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981140    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981144    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981146    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981149    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981153    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981158    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981162    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981164    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981167    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981170    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981173    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981175    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981178    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981181    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981184    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:01.981192 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981189    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981192    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981195    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981198    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981201    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981204    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981207    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981210    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981212    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981215    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981218    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981221    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981223    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981239    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981242    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981245    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981248    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981250    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981253    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981256    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:01.981716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981259    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981261    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981264    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981267    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981271    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981275    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981279    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981282    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981285    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981287    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981290    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981293    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981296    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981298    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981301    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981303    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981306    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981314    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981317    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:01.982218 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981320    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981322    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981325    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981328    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981330    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981333    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981336    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981338    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981341    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981343    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981346    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981349    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981351    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981354    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981357    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981360    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981362    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981365    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981368    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981371    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:01.982751 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981373    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981376    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981379    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981382    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981385    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981388    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981390    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981393    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981396    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981398    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981401    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981404    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981406    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981409    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981411    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981414    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981417    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981419    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981422    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:01.983260 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981424    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981428    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981430    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981433    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981436    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981439    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981441    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.981444    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982217    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982245    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982249    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982254    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982257    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982260    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982263    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982266    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982270    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982272    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982275    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:01.983739 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982278    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982281    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982284    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982287    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982289    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982292    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982294    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982297    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982299    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982302    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982304    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982307    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982310    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982312    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982315    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982317    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982320    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982322    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982325    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982328    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:01.984217 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982330    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982333    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982336    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982339    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982342    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982345    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982348    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982350    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982353    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982357    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982359    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982362    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982365    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982368    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982372    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982375    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982378    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982380    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982383    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:01.984722 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982385    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982388    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982391    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982394    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982396    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982399    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982401    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982404    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982406    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982409    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982411    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982414    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982417    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982419    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982421    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982424    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982427    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982429    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982432    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982435    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:01.985199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982438    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982441    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982444    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982446    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982449    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982451    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982454    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982457    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982460    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982463    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982465    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982468    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982472    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982476    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982479    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982482    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982485    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982488    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982490    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:01.985704 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982493    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982495    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982498    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982500    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982503    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982505    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982508    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982511    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982513    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982516    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982519    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982522    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982524    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982527    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982529    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982533    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.982536    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983014    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983019    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983023    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:01.986282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983026    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983029    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983032    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983034    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983037    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983040    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983042    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983045    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983048    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983051    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983053    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983056    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983058    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983061    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983063    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983066    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983068    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983071    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983073    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:01.986837 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983076    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983078    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983081    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983084    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983088    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983091    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983093    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983096    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983098    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983101    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983104    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983107    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983109    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983112    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983114    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983117    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983119    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983122    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983124    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983127    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:01.987651 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983129    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983132    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983134    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983137    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983139    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983142    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983144    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983147    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983149    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983152    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983155    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983157    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983160    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983163    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983165    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983168    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983171    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983173    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983176    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983178    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:01.988383 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983181    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983183    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983186    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983189    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983191    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983194    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983196    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983199    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983202    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983204    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983207    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983209    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983211    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983214    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983216    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983219    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983221    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983224    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983240    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983242    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:01.988990 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983245    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983248    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983250    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983253    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983255    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983258    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983261    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983264    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983267    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983269    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983272    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983276    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983281    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983285    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983288    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983291    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983294    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983297    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983300    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:01.989526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983303    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983308    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983312    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983315    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.983318    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983411    6481 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983425    6481 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983431    6481 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983435    6481 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983440    6481 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983443    6481 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983450    6481 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983454    6481 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983457    6481 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983461    6481 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983464    6481 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983467    6481 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983471    6481 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983474    6481 flags.go:64] FLAG: --cgroup-root=""
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983477    6481 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983480    6481 flags.go:64] FLAG: --client-ca-file=""
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983483    6481 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983486    6481 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 14:12:01.990007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983490    6481 flags.go:64] FLAG: --cluster-domain=""
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983493    6481 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983496    6481 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983499    6481 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983502    6481 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983506    6481 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983509    6481 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983512    6481 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983515    6481 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983518    6481 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983521    6481 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983524    6481 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983528    6481 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983531    6481 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983535    6481 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983538    6481 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983541    6481 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983544    6481 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983547    6481 flags.go:64] FLAG: --enable-server="true"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983549    6481 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983555    6481 flags.go:64] FLAG: --event-burst="100"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983560    6481 flags.go:64] FLAG: --event-qps="50"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983563    6481 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983567    6481 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 14:12:01.990589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983570    6481 flags.go:64] FLAG: --eviction-hard=""
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983575    6481 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983578    6481 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983581    6481 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983584    6481 flags.go:64] FLAG: --eviction-soft=""
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983587    6481 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983590    6481 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983593    6481 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983597    6481 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983600    6481 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983603    6481 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983606    6481 flags.go:64] FLAG: --feature-gates=""
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983610    6481 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983613    6481 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983616    6481 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983620    6481 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983623    6481 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983626    6481 flags.go:64] FLAG: --help="false"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983629    6481 flags.go:64] FLAG: --hostname-override="ip-10-0-128-190.ec2.internal"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983632    6481 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983635    6481 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983638    6481 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983642    6481 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 14:12:01.991174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983645    6481 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983648    6481 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983651    6481 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983654    6481 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983657    6481 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983661    6481 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983666    6481 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983669    6481 flags.go:64] FLAG: --kube-reserved=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983672    6481 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983675    6481 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983678    6481 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983681    6481 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983684    6481 flags.go:64] FLAG: --lock-file=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983687    6481 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983690    6481 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983693    6481 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983698    6481 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983701    6481 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983705    6481 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983708    6481 flags.go:64] FLAG: --logging-format="text"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983711    6481 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983715    6481 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983717    6481 flags.go:64] FLAG: --manifest-url=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983720    6481 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983725    6481 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 14:12:01.991753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983728    6481 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983732    6481 flags.go:64] FLAG: --max-pods="110"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983735    6481 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983738    6481 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983741    6481 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983744    6481 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983748    6481 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983751    6481 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983754    6481 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983761    6481 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983764    6481 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983767    6481 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983771    6481 flags.go:64] FLAG: --pod-cidr=""
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983776    6481 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983779    6481 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983782    6481 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983785    6481 flags.go:64] FLAG: --port="10250"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983788    6481 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983791    6481 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-053ceb1ac93318e17"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983794    6481 flags.go:64] FLAG: --qos-reserved=""
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983798    6481 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983801    6481 flags.go:64] FLAG: --register-node="true"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983804    6481 flags.go:64] FLAG: --register-with-taints=""
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983808    6481 flags.go:64] FLAG: --registry-burst="10"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983811    6481 flags.go:64] FLAG: --registry-qps="5"
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983814    6481 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 14:12:01.992389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983817    6481 flags.go:64] FLAG: --reserved-memory=""
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983821    6481 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983824    6481 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983827    6481 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983830    6481 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983833    6481 flags.go:64] FLAG: --runonce="false"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983836    6481 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983839    6481 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983842    6481 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983846    6481 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983848    6481 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983851    6481 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983854    6481 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983858    6481 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983860    6481 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983864    6481 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983866    6481 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983869    6481 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983874    6481 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983880    6481 flags.go:64] FLAG: --system-cgroups=""
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983883    6481 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983889    6481 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983892    6481 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983894    6481 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983898    6481 flags.go:64] FLAG: --tls-min-version=""
Nov 03 14:12:01.993152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983901    6481 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983904    6481 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983907    6481 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983909    6481 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983912    6481 flags.go:64] FLAG: --v="2"
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983917    6481 flags.go:64] FLAG: --version="false"
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983921    6481 flags.go:64] FLAG: --vmodule=""
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983925    6481 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.983928    6481 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984065    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984069    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984072    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984074    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984077    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984080    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984083    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984086    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984088    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984091    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984094    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984096    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:01.993792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984099    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984102    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984104    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984107    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984111    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984115    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984117    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984120    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984123    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984125    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984128    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984131    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984133    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984136    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984139    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984141    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984144    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984147    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984149    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:01.994630 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984152    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984155    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984157    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984160    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984162    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984165    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984167    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984170    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984172    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984175    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984178    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984180    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984183    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984186    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984190    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984193    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984195    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984200    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984204    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:01.995309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984206    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984209    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984212    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984214    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984218    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984220    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984223    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984238    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984241    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984244    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984247    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984249    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984253    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984257    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984260    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984263    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984266    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984269    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984271    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:01.995848 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984274    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984277    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984279    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984282    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984285    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984288    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984290    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984293    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984295    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984298    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984300    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984304    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984308    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984311    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984314    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984316    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984319    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984322    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984324    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:01.996457 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984327    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984330    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984332    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984335    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984337    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984340    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984342    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984345    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984348    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984350    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984353    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984356    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984359    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984361    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984364    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984366    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984369    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:01.996969 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984372    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.984396    6481 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984437    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984442    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984445    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984448    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984451    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984456    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984461    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984465    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984468    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984472    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984474    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984477    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984479    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984482    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:01.997411 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984485    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984487    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984490    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984492    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984495    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984498    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984500    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984503    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984505    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984509    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984513    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984517    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984520    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984522    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984525    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984527    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984530    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984533    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984535    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:01.997885 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984538    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984541    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984544    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984546    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984550    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984553    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984557    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984559    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984563    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984566    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984568    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984571    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984574    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984576    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984579    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984581    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984584    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984587    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984589    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984592    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:01.998603 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984594    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984597    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984599    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984602    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984605    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984607    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984610    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984612    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984615    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984617    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984620    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984622    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984625    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984628    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984630    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984633    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984637    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984639    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984643    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984646    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:01.999152 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984648    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984651    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984654    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984656    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984659    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984662    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984664    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984667    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984669    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984672    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984675    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984677    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984680    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984682    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984685    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984688    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984690    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984694    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984696    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:01.999657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984699    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984702    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984704    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984707    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984709    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984712    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984714    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984717    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984719    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984724    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984727    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984731    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984734    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.984737    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.990062    6481 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.990077    6481 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.990101    6481 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.990106    6481 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990173    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:02.000121 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990178    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990181    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990184    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990188    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990190    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990193    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990196    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990199    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990201    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990204    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990207    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990210    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990213    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990216    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990219    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990221    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990224    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990261    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990265    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:02.000662 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990269    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990273    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990277    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990280    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990282    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990285    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990288    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990291    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990294    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990297    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990300    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990303    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990305    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990308    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990311    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990313    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990316    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990318    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990321    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990324    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:02.001120 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990326    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990329    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990332    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990334    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990337    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990339    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990343    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990346    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990348    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990351    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990353    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990356    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990359    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990361    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990364    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990366    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990369    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990371    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990374    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990377    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:02.001613 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990380    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990383    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990385    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990388    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990391    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990393    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990396    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990399    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990401    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990404    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990406    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990409    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990411    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990414    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990417    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990419    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990422    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990425    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990427    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990430    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:02.002106 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990433    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990436    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990439    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990441    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990444    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990446    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990449    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990452    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990455    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990457    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990460    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990462    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990465    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990468    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990470    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990473    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990475    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990478    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990480    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990483    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:02.002616 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990486    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990488    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990491    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990495    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990499    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990501    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.990530    6481 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990677    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990682    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990685    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990688    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990691    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990694    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990697    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990700    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:02.003272 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990703    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990705    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990708    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990711    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990713    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990716    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990719    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990722    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990724    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990727    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990729    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990732    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990735    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990737    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990740    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990742    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990745    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990747    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990750    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990752    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:02.003878 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990755    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990758    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990761    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990763    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990766    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990769    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990772    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990774    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990777    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990779    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990783    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990785    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990788    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990790    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990793    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990795    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990798    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990801    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990803    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:02.004712 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990806    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990808    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990811    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990813    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990816    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990819    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990821    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990824    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990826    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990829    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990831    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990834    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990837    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990839    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990842    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990845    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990848    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990850    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990853    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990856    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:02.005199 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990858    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990861    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990863    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990866    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990869    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990872    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990875    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990877    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990880    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990882    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990885    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990887    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990890    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990892    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990895    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990897    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990901    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990905    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990909    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:02.006045 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990912    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990916    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990919    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990921    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990924    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990927    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990929    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990932    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990935    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990938    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990940    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990943    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990945    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990948    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990951    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990953    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990956    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990959    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990962    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:02.006797 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.990964    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.990988    6481 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991019    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991024    6481 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991027    6481 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991030    6481 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991032    6481 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991035    6481 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991038    6481 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991041    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991044    6481 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991046    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991049    6481 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991051    6481 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991054    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991060    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 14:12:02.007406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991063    6481 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991066    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991069    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991071    6481 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991074    6481 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991077    6481 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991079    6481 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991082    6481 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991084    6481 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991087    6481 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991090    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991093    6481 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991095    6481 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991098    6481 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991100    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991103    6481 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991106    6481 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991109    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991112    6481 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 14:12:02.007807 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991114    6481 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991117    6481 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991120    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991122    6481 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991125    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991127    6481 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991130    6481 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991132    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991135    6481 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991138    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991140    6481 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991143    6481 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991147    6481 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991149    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991152    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991155    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991157    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991160    6481 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991163    6481 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 14:12:02.008277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991165    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991168    6481 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991171    6481 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991173    6481 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991176    6481 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991179    6481 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991182    6481 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991184    6481 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991187    6481 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991190    6481 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991193    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991196    6481 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991199    6481 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991201    6481 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991204    6481 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991207    6481 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991209    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991213    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991215    6481 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991218    6481 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 14:12:02.008774 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991221    6481 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991240    6481 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991244    6481 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991247    6481 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991250    6481 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991254    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991257    6481 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991261    6481 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991265    6481 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991268    6481 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991271    6481 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991275    6481 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991278    6481 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991281    6481 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991284    6481 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991287    6481 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991289    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991292    6481 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991295    6481 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 14:12:02.009275 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991298    6481 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991301    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991304    6481 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991306    6481 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991309    6481 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991312    6481 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991315    6481 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991317    6481 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991320    6481 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991323    6481 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991325    6481 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991328    6481 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991330    6481 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991333    6481 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:01.991340    6481 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.991390    6481 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.992325    6481 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.992376    6481 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 14:12:02.009732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.992577    6481 server.go:1014] "Starting client certificate rotation"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.992725    6481 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.992788    6481 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 14:05:50 +0000 UTC" deadline="2028-08-02 21:12:14.12762029 +0000 UTC"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.992830    6481 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15319h0m12.134793509s"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.993050    6481 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.994094    6481 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.995971    6481 log.go:25] "Validated CRI v1 runtime API"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.997887    6481 log.go:25] "Validated CRI v1 image API"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:01.998701    6481 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 14:12:02.010189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.000976    6481 fs.go:135] Filesystem UUIDs: map[73e197d9-c4a4-4e87-b00e-64552d1e35da:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c6f67808-7d54-469f-8aa0-a28cb27b0a11:/dev/nvme0n1p3]
Nov 03 14:12:02.010466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.000996    6481 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae/userdata/shm major:0 minor:276 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc/userdata/shm major:0 minor:434 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf/userdata/shm major:0 minor:309 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6/userdata/shm major:0 minor:429 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363/userdata/shm major:0 minor:431 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~projected/kube-api-access-d8729:{mountpoint:/var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~projected/kube-api-access-d8729 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~projected/kube-api-access-7kmsg:{mountpoint:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~projected/kube-api-access-7kmsg major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d998c6-eb92-4c4e-b35e-fa3703de95ed/volumes/kubernetes.io~projected/kube-api-access-tjjw8:{mountpoint:/var/lib/kubelet/pods/15d998c6-eb92-4c4e-b35e-fa3703de95ed/volumes/kubernetes.io~projected/kube-api-access-tjjw8 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15d998c6-eb92-4c4e-b35e-fa3703de95ed/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/15d998c6-eb92-4c4e-b35e-fa3703de95ed/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/167d067e-3824-4add-b57b-9ba2bfc6c200/volumes/kubernetes.io~projected/kube-api-access-7zlms:{mountpoint:/var/lib/kubelet/pods/167d067e-3824-4add-b57b-9ba2bfc6c200/volumes/kubernetes.io~projected/kube-api-access-7zlms major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb511bd-1e7a-44c1-b027-964d3fb57d08/volumes/kubernetes.io~projected/kube-api-access-2dncp:{mountpoint:/var/lib/kubelet/pods/1eb511bd-1e7a-44c1-b027-964d3fb57d08/volumes/kubernetes.io~projected/kube-api-access-2dncp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3196f084-b0f3-4145-be7f-0d2b63dd8019/volumes/kubernetes.io~projected/kube-api-access-7h9mw:{mountpoint:/var/lib/kubelet/pods/3196f084-b0f3-4145-be7f-0d2b63dd8019/volumes/kubernetes.io~projected/kube-api-access-7h9mw major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3196f084-b0f3-4145-be7f-0d2b63dd8019/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3196f084-b0f3-4145-be7f-0d2b63dd8019/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40daba9a-1280-486d-b7f4-047d0efdab2e/volumes/kubernetes.io~projected/kube-api-access-blgnm:{mountpoint:/var/lib/kubelet/pods/40daba9a-1280-486d-b7f4-047d0efdab2e/volumes/kubernetes.io~projected/kube-api-access-blgnm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40daba9a-1280-486d-b7f4-047d0efdab2e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/40daba9a-1280-486d-b7f4-047d0efdab2e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/config-reloader/2 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/prometheus/15 major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~projected/kube-api-access-95stb:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~projected/kube-api-access-95stb major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~projected/tls-assets major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/config major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/web-config major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8edbb9a8-e48f-428f-9967-61dd8333133e/volumes/kubernetes.io~projected/kube-api-access-qtkpv:{mountpoint:/var/lib/kubelet/pods/8edbb9a8-e48f-428f-9967-61dd8333133e/volumes/kubernetes.io~projected/kube-api-access-qtkpv major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9225fde2-c7af-4b4e-80c0-6a473f655fec/volumes/kubernetes.io~projected/kube-api-access-mbssx:{mountpoint:/var/lib/kubelet/pods/9225fde2-c7af-4b4e-80c0-6a473f655fec/volumes/kubernetes.io~projected/kube-api-access-mbssx major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9225fde2-c7af-4b4e-80c0-6a473f655fec/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9225fde2-c7af-4b4e-80c0-6a473f655fec/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9713304b-d13e-4e2c-b460-6a02c9adf5b4/volumes/kubernetes.io~projected/kube-api-access-f9ns6:{mountpoint:/var/lib/kubelet/pods/9713304b-d13e-4e2c-b460-6a02c9adf5b4/volumes/kubernetes.io~projected/kube-api-access-f9ns6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9927b257-7027-4ef3-aa19-bde73b4d93b9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/9927b257-7027-4ef3-aa19-bde73b4d93b9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4fa7bf3-f448-451d-b46e-a885de36da57/volumes/kubernetes.io~projected/kube-api-access-xc45c:{mountpoint:/var/lib/kubelet/pods/a4fa7bf3-f448-451d-b46e-a885de36da57/volumes/kubernetes.io~projected/kube-api-access-xc45c major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1c76fe7-8625-4a41-a039-af53c4978a94/volumes/kubernetes.io~projected/kube-api-access-fr6tm:{mountpoint:/var/lib/kubelet/pods/b1c76fe7-8625-4a41-a039-af53c4978a94/volumes/kubernetes.io~projected/kube-api-access-fr6tm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6e29be6-c48c-4bc8-a601-a0e337715b61/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d6e29be6-c48c-4bc8-a601-a0e337715b61/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6e35132-a355-4ec6-a6f6-3ef83c5c2840/volumes/kubernetes.io~projected/kube-api-access-mpfvh:{mountpoint:/var/lib/kubelet/pods/d6e35132-a355-4ec6-a6f6-3ef83c5c2840/volumes/kubernetes.io~projected/kube-api-access-mpfvh major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~projected/kube-api-access-wh7vl:{mountpoint:/var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~projected/kube-api-access-wh7vl major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volumes/kubernetes.io~projected/kube-api-access-65rvz:{mountpoint:/var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volumes/kubernetes.io~projected/kube-api-access-65rvz major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2d69636-c98b-4456-93be-168d230451b6/volumes/kubernetes.io~projected/kube-api-access-9xxbk:{mountpoint:/var/lib/kubelet/pods/f2d69636-c98b-4456-93be-168d230451b6/volumes/kubernetes.io~projected/kube-api-access-9xxbk major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2d69636-c98b-4456-93be-168d230451b6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f2d69636-c98b-4456-93be-168d230451b6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~projected/kube-api-access-46t45:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~projected/kube-api-access-46t45 major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/alertmanager/9 major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/config-reloader/6 major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~projected/kube-api-access-67zb7:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~projected/kube-api-access-67zb7 major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~projected/tls-assets major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/config-volume major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/web-config major:0 minor:251 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/552d5f3a6e0625d4e676097d0909e933bf4d2765a1bc145cadb16852aa86f059/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9022794c41fdb26a2345970432c7e75379c12dd81698b0318442c7a333424706/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/352bc5c10cfe134b6091e6d2ac045f0929bb3762e3c5bf35e17fbe02413bf300/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/1d88b7889469030e22543217c7a9db78fbc6b5f1f727238b331dc9ab4cfff4ac/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d30575df375e9ae9b5cac3039b64dbdcf79583251b22534a3da3bebd68f33a40/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/95498239189d99f864849f46c3e87949a12e147e559d2050e557dc4df29fd270/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/465775157cd8d6a812341512e11fcf125f5ee0be7ef0d66a2e21a8226ad06805/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/714f6dcfabf53f6821c099c92d9944babe35672d1d980caba27d7edbd6033718/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b3221722a4ad6f98a5c8ff642d9015c6a1e826f290f2321100244e7b52728ce5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7cf8617335db1f07a10ba345581bb321c55982f1c2417936967b11fbef138d0a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/64582c10f3908a8e16b92c3060d9ee511d5ef81a13eae77463910eae5e00c64b/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/422eaf2e28943c6d44a1e589b70b212f0831c3d5c74b570bd3c471c1a024ad73/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5a7deda7ab8a6dc7d9199267f43d1a7a7eacccdef2cf3908c8d8e406c033fe36/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/b15463583c94137d1b7d63ebcaca30edf9c44a2d965b32c9823bcf5dbba5e4d4/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/fa3e7719bfbe20dc4d1ebd7908a3332d87e50684cbafbca6ad58db04aa9503ce/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/324038aa2ce0f44405faf11c4c63c2380607b15a0f067212a40666d7e8a3acbd/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8f19b91fce40e3af34b2e404396de5b0b8f9abf9e454699b39d84fffa3dd5941/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4bb8195f16f9a24d070b6cf66e92f37a0e7cb37b8275276940deb012366dae08/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/607175e54122fe75316d98c9afedc07d2f04363ab5189650b048208d5105f11b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5f209ca7ea21b4ca6fde439ea9f400efaf6392451f276c6f97c0a9333616de81/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d5c3872b4bf30887fc965bff157e675ceafbfe84212efc07d74470fa4e962479/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/5fe50e8ce40743174200b0ef18d540ac3f908f4892e986750c4d36de167a7339/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/fbaa140da4107253fd3c6f55988f7aa148f86b8a6bd0fd541b21c3995a2309f5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/2d5fb4b10781d5fb05a4bc1ac3f0419b656143b7b5bc2251f1e10c2c5bdd4009/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/a0c053f098f97669dc2d04b55ce592600d418fe754f7f9ecbac442a15b200cfb/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/094d0227d0bdaaa53882659b579706fc043c32b814dbc4089bd65b24401ead9d/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/510805e3e1d82f603b4367907354776c00e18b063651397f31163e41851a02db/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/f52b64423b8945e9340f3893291ea172aa00753007ebefa83f65cee4dda356d0/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/41d23e6a6d005da7fd6df27ebdef1c280562982b881598dfb1ba5ee1bbcd8584/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/b1944a2ad4526438624c633234520a77799611d887528664195a26fbd7b5afab/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/3ac29a336bb8ca93a97a9d2bc359b3dfc68b87aecab75aadad254b9b282c43d1/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/d700c8392c6458be6a0a9eb30de0b50f0a52425afeb9b78995b9600fb6a2c7c3/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/6af23b7c1be801dba1aa56745513929c5f551a6d292a68def1789ceb9589dbf9/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/91f17bcee5f40d5a108a66da6f29315b0d61f4080958913917e97cef0b632cd3/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/5358d7f2a760469b82d400b54444682a86654c5b69301cd829f0b8d6834a31c9/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/881402a11c3c4284f0bf9784b863e23d2b122c47d60c58f86fd1421672deed8d/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/af7941bf63917ad6a5c3f8c639d0daa456d58692ca648223bbd40c0bef136b80/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/5cb2adec0aca43934a3fad06f568b7d2a3292c4fbc98274551a20adf90a900a0/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/21e679bdce912ff508832c32209f9b750f4f9da6405dcc6d1eaf0dbd863c6016/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/304f52d807d4422c745e786fe6fa5d8a40910eca4a5d2b42d62360fd87506f43/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a13064e08f2d2d94c1e663a6e23c713b0da43b045d5d731cfcbe8885900209cb/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/7177746b82a19b09ef06527eb2494ade2ac005397078169a6b2fcf57fe90e0b2/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/4567c10427d20948b44024c5a61f2c1df66e4325e70ec5b3d01f98c87a7d8ba5/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/810d073a6ed3296a2bbdd64c1d4065a11f650e35af6026764ddfa4dc9fd1728b/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/6cbb58899db6a6dfc7d158c076451a45d47b1415fccab784060b00bd2c3309e5/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/8e57f67a0f6e85bc440f3ad6b5eb4ddc8af0a7419f3bf319623631b017e745f5/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/10db6528b5725259d5578726c1537744f1dd4e99a602a76db4405874f2a4b902/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/21a33e401a1dd003a627ea931fe85ce06ff0798565f838d3733be6c34a394dc9/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/9467d0de3ab4d4c1a1bcc66beb7b8fbf4572d6857467ae4063d8c20597b00541/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/dd0a523665b62c04cab40b0cccfb9dd95d685d87cc5bd215ddf74eae8363d86e/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/fc6b562f8d50e6facec47ee9ad01d583699e1d1677ae62033b42b914ec89c9cd/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/8adb9f0c6b578867ebfdfb56ee77c9052885780f0d1c74acf17e986dfad80d2d/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/ca00dd1c09d491b20b1b86d580f9ca2a924a22faf60ed4d683a5efb7ca5c58c7/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/eb1c97b9e69e1eaf993f2ad98adf30542a4086064eadc4766b614161b84fd7e2/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/cc314225cca5bceb52543f48a045ede80630a10bd90d151876edb939d0f916be/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/0e3bc2de7d3359ebeee40263993623ad6ae8620f0d57845fe5dd8de70f736ce3/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/92836f2ac67bc4b21cd85fff253bca1aeeafbd542ccaf7a9db0c6cb39eb9ab78/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/e6b476b6795543ba89c073885110ebc07ef473b5383654ab5bd7f44edec81493/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/7f78748c52d03da979e827d923f2bbd90c15790b066fbc4028fc64c3e0745b2b/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/a502feee43d29afe35744a7834f9d833a355ee8bfa7dd12b7f0b29955ba7c65d/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/ebc5e50496fca856a88e03da1d1470b283945d22043af95c0feb24e9448359ee/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/f7a6cf86bf11b50d24ce12c61fb546bab29fa01f3a6f4c7a09d404d5b4ae6d50/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/385dc7c2fb299e7d42223be2017f8d58ac64bb1922e63d2091114ca18a3b84a6/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/c2f116155cd59cd6f2fb4f9fe4257adf08f13471aafa0b1f3a9cbb33c3a02b08/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/ca85f1ab06388053018fea5ee893d16cccdb33e3b9e0103dd176843aac5e9a88/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/de48f390623e3e35699b6be73b2ea5347a9da06830862b6db2ff31605ca88d8a/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3655bda7c627cec0aee30f66bdd2dc3d975607a044b82278263cf1b4d481719d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5e71af71054a18d390d5d91a94c78a62c0b8d5106b6397c468d52c8a133d83dd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7bccb8d194e7181ce169a31b26af5e1458160e16728b4508fb5c125d44e9e3f3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/264cd58ff93fbab8d14db2adb96623dfaf0f4d3f85632ef7445f741fe5f95c19/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6bbd4015f6bd0f0300473492fe00b918236e75284ef0d31ee911d963d7f35760/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/7a40d25fce359d8b21157d30fdaf636b41c62b2c9c8c404cb061a25b19b1d005/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/08d8067a7b09b2dcf8cfa95adf3a931bd264b0c24fd837118dbb79689ab43bf7/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/b091d6f649791bc8af2eb0e170843c95ef4af11b205b0f8074d173bc17fa2713/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4580616d4f86b2d863e6d77d1afe50ea5a2026b5a56e4387b845a9f36d922522/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.009589    6481 manager.go:217] Machine: {Timestamp:2026-11-03 14:12:02.009104862 +0000 UTC m=+0.059136037 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3066837 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28f369d823326f19dc633f4229cbde SystemUUID:ec28f369-d823-326f-19dc-633f4229cbde BootID:bde9a5c6-48e4-43f8-a770-a3b57f8f45d5 Filesystems:[{Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84058db707444d9f307c86e2584d9383f5b47487d69026b3d2aefc3ded8ff1d4/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:354 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~projected/kube-api-access-d8729 DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e90b2b3c9ad6f0317041a7ac25e7a401b246a1f05385c3be29c77aa67bb1f15/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~projected/kube-api-access-95stb DeviceMajor:0 DeviceMinor:360 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5acb1c0d093cd91d94ea8da332a4aae14a6cec76d1d653c60df3867d491849bc/userdata/shm DeviceMajor:0 DeviceMinor:434 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/475188a3aa07a721a5f429ab69cdec13da6266ef8b4aeaeb9ce69c3fe69d6817/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ef8c40fc28e051576e7b9f32a8bd2711cb13f78ef17e18485e52b82b19597af/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6e35132-a355-4ec6-a6f6-3ef83c5c2840/volumes/kubernetes.io~projected/kube-api-access-mpfvh DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2d69636-c98b-4456-93be-168d230451b6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f5e833ccdb29505d17f4d92d2e4140093595c20cbd2ff755fdc4fcc1101cb2d1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:257 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2d69636-c98b-4456-93be-168d230451b6/volumes/kubernetes.io~projected/kube-api-access-9xxbk DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85c089ef988b796a528dc498730432073c18abcd4962a181842bdcb7c92b75d7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~projected/kube-api-access-wh7vl DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1c76fe7-8625-4a41-a039-af53c4978a94/volumes/kubernetes.io~projected/kube-api-access-fr6tm DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~projected/kube-api-access-67zb7 DeviceMajor:0 DeviceMinor:256 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:345 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4a8f8b020f3a1e948751de4d305e4a4a16a7405c8dd2ab5e027038ae6fb4f13/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9713304b-d13e-4e2c-b460-6a02c9adf5b4/volumes/kubernetes.io~projected/kube-api-access-f9ns6 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8b280d118fb113e86485ae1ff7c5bc2d7744d2d555b085be41b8de4ac6b52b0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15d998c6-eb92-4c4e-b35e-fa3703de95ed/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:259 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9927b257-7027-4ef3-aa19-bde73b4d93b9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:428 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:261 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:357 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cde2d973a536953d6e951ab58860c292bc8a59751ed31db705c563de7033e45c/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:352 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:258 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:260 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:289 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9225fde2-c7af-4b4e-80c0-6a473f655fec/volumes/kubernetes.io~projected/kube-api-access-mbssx DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76d705ce335979ebe20d483d26bb58dc59ab5d4685054bf4bddf2a3631963fda/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d836117c6d4c3f5fc1affb56b7114538988c7e5883d34f1c71b4b13f13a3f4a6/userdata/shm DeviceMajor:0 DeviceMinor:429 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8edbb9a8-e48f-428f-9967-61dd8333133e/volumes/kubernetes.io~projected/kube-api-access-qtkpv DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40daba9a-1280-486d-b7f4-047d0efdab2e/volumes/kubernetes.io~projected/kube-api-access-blgnm DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b67e775285eb99ceb21f1d717c9e385fcd5270379297b9bff5b4c05255a44661/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:350 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:353 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d6e29be6-c48c-4bc8-a601-a0e337715b61/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da81e815f57a78d29141cefeec448d0ec1a6c1d8b264de052d1069289dac6363/userdata/shm DeviceMajor:0 DeviceMinor:431 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/volumes/kubernetes.io~projected/kube-api-access-46t45 DeviceMajor:0 DeviceMinor:262 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3196f084-b0f3-4145-be7f-0d2b63dd8019/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:351 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12aceb0ba5e63e6fbf3fe94014ab132edbadf85196f7cd028bfeb6b4d456d0ae/userdata/shm DeviceMajor:0 DeviceMinor:276 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9cfec689f3af932846bb8040260dc388486dffa9398d18d769011ed77dbbbfcf/userdata/shm DeviceMajor:0 DeviceMinor:309 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:359 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/167d067e-3824-4add-b57b-9ba2bfc6c200/volumes/kubernetes.io~projected/kube-api-access-7zlms DeviceMajor:0 DeviceMinor:433 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1eb511bd-1e7a-44c1-b027-964d3fb57d08/volumes/kubernetes.io~projected/kube-api-access-2dncp DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/15d998c6-eb92-4c4e-b35e-fa3703de95ed/volumes/kubernetes.io~projected/kube-api-access-tjjw8 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:358 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0977c82f-e896-4817-b4bf-eb1b056060c6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a4fa7bf3-f448-451d-b46e-a885de36da57/volumes/kubernetes.io~projected/kube-api-access-xc45c DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b37b238abe89d5aa506ed4da5d858c684a0809b560cc560eaea13d3a6555e01e/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:244 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volumes/kubernetes.io~projected/kube-api-access-65rvz DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9225fde2-c7af-4b4e-80c0-6a473f655fec/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ccb6ab-b665-4753-96a2-9ce1fa145666/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40daba9a-1280-486d-b7f4-047d0efdab2e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:424 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3196f084-b0f3-4145-be7f-0d2b63dd8019/volumes/kubernetes.io~projected/kube-api-access-7h9mw DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:355 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:356 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0bc6d4db-06d7-4767-bef1-d90cfb51dc88/volumes/kubernetes.io~projected/kube-api-access-7kmsg DeviceMajor:0 DeviceMinor:291 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96ea693abe196c5a2d5aa3a2fc0b4dd4dcdec0a3a503eff28276d5e1dd83fd9f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb21c178-123e-4c5c-a0f3-e59e5aab8927/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:12aceb0ba5e63e6 MacAddress:62:da:03:83:a6:d0 Speed:10000 Mtu:8901} {Name:457b7000c7de7bc MacAddress:52:3c:d7:d4:7f:8b Speed:10000 Mtu:8901} {Name:5acb1c0d093cd91 MacAddress:d2:66:60:6c:ae:3a Speed:10000 Mtu:8901} {Name:76d705ce335979e MacAddress:2a:aa:86:a7:ff:12 Speed:10000 Mtu:8901} {Name:7dfd1c0e31c9446 MacAddress:ae:c4:9a:b8:4c:a1 Speed:10000 Mtu:8901} {Name:7ef8c40fc28e051 MacAddress:26:a1:f6:13:b9:54 Speed:10000 Mtu:8901} {Name:84058db707444d9 MacAddress:f6:d3:18:11:b4:09 Speed:10000 Mtu:8901} {Name:9cfec689f3af932 MacAddress:4e:1f:6f:a2:80:bc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f4:bc:f6:67 Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:9a:f5:6a:47:62 Speed:0 Mtu:8901} {Name:d836117c6d4c3f5 MacAddress:b2:a4:fe:19:03:75 Speed:10000 Mtu:8901} {Name:da81e815f57a78d MacAddress:ce:3f:c5:67:5f:16 Speed:10000 Mtu:8901} {Name:e4a8f8b020f3a1e MacAddress:3e:1a:70:85:69:02 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f4:bc:f6:67 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:a1:e4:3c:c2:a4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:f4:03:54:e2:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010080    6481 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010125    6481 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010457    6481 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010483    6481 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010608    6481 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010615    6481 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010625    6481 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010637    6481 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 14:12:02.010725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010715    6481 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010867    6481 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.010973    6481 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.011008    6481 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.011023    6481 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.011032    6481 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.011041    6481 kubelet.go:405] "Adding apiserver pod source"
Nov 03 14:12:02.011207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.011049    6481 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 14:12:02.012067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.012049    6481 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 14:12:02.012067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.012067    6481 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 14:12:02.012586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.012566    6481 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 14:12:02.015168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015152    6481 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 14:12:02.015168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015168    6481 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 14:12:02.015293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015277    6481 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 14:12:02.015488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015476    6481 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 14:12:02.015522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015506    6481 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 14:12:02.015626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015616    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 14:12:02.015658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015632    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 14:12:02.015658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015638    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 14:12:02.015658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015644    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015666    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015674    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015679    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015685    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015691    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015697    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015709    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 14:12:02.015734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015719    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 14:12:02.015978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015742    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 14:12:02.015978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.015751    6481 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 14:12:02.019039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.019027    6481 server.go:1263] "Started kubelet"
Nov 03 14:12:02.019198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.019120    6481 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 14:12:02.019276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.019244    6481 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 14:12:02.019276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.019258    6481 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 14:12:02.020673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.020566    6481 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 14:12:02.019608 ip-10-0-128-190 systemd[1]: Started Kubernetes Kubelet.
Nov 03 14:12:02.021559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.021536    6481 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 14:12:02.025988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.025498    6481 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 14:12:02.025988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.025511    6481 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 14:12:02.025988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.025599    6481 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 14:05:52 +0000 UTC" deadline="2028-08-16 20:29:27.883779866 +0000 UTC"
Nov 03 14:12:02.025988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.025639    6481 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15654h17m25.858144408s"
Nov 03 14:12:02.025988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.025810    6481 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 14:12:02.025988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.025838    6481 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 14:12:02.026792 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.026771    6481 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-190.ec2.internal\" not found"
Nov 03 14:12:02.026841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.026827    6481 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 14:12:02.028066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.027793    6481 factory.go:55] Registering systemd factory
Nov 03 14:12:02.028066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.027814    6481 factory.go:223] Registration of the systemd container factory successfully
Nov 03 14:12:02.028212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028072    6481 factory.go:153] Registering CRI-O factory
Nov 03 14:12:02.028212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028088    6481 factory.go:223] Registration of the crio container factory successfully
Nov 03 14:12:02.028212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028138    6481 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 14:12:02.028212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028159    6481 factory.go:103] Registering Raw factory
Nov 03 14:12:02.028212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028183    6481 manager.go:1199] Started watching for new ooms in manager
Nov 03 14:12:02.028568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028553    6481 manager.go:319] Starting recovery of all containers
Nov 03 14:12:02.029009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.028977    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029014    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029030    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029042    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ccb6ab-b665-4753-96a2-9ce1fa145666" volumeName="kubernetes.io/empty-dir/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-textfile" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029055    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d69636-c98b-4456-93be-168d230451b6" volumeName="kubernetes.io/projected/f2d69636-c98b-4456-93be-168d230451b6-kube-api-access-9xxbk" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029066    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/configmap/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-metrics-client-ca" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029079    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-grpc-tls" seLinuxMountContext=""
Nov 03 14:12:02.029101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029090    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029105    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0977c82f-e896-4817-b4bf-eb1b056060c6" volumeName="kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-tmp" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029117    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3196f084-b0f3-4145-be7f-0d2b63dd8019" volumeName="kubernetes.io/secret/3196f084-b0f3-4145-be7f-0d2b63dd8019-metrics-tls" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029129    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029141    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029154    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9927b257-7027-4ef3-aa19-bde73b4d93b9" volumeName="kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029166    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e29be6-c48c-4bc8-a601-a0e337715b61" volumeName="kubernetes.io/secret/d6e29be6-c48c-4bc8-a601-a0e337715b61-agent-certs" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029180    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb21c178-123e-4c5c-a0f3-e59e5aab8927" volumeName="kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-env-overrides" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029191    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d69636-c98b-4456-93be-168d230451b6" volumeName="kubernetes.io/empty-dir/f2d69636-c98b-4456-93be-168d230451b6-data-volume" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029206    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029219    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" volumeName="kubernetes.io/projected/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-kube-api-access-mpfvh" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029244    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d69636-c98b-4456-93be-168d230451b6" volumeName="kubernetes.io/secret/f2d69636-c98b-4456-93be-168d230451b6-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029256    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029268    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029279    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-client-ca-bundle" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029293    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3196f084-b0f3-4145-be7f-0d2b63dd8019" volumeName="kubernetes.io/configmap/3196f084-b0f3-4145-be7f-0d2b63dd8019-config-volume" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029306    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40daba9a-1280-486d-b7f4-047d0efdab2e" volumeName="kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029320    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029332    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029344    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029356    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029383    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029396    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="167d067e-3824-4add-b57b-9ba2bfc6c200" volumeName="kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029411    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029425    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029436    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8edbb9a8-e48f-428f-9967-61dd8333133e" volumeName="kubernetes.io/configmap/8edbb9a8-e48f-428f-9967-61dd8333133e-serviceca" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029448    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1c76fe7-8625-4a41-a039-af53c4978a94" volumeName="kubernetes.io/projected/b1c76fe7-8625-4a41-a039-af53c4978a94-kube-api-access-fr6tm" seLinuxMountContext=""
Nov 03 14:12:02.029450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029459    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029470    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d998c6-eb92-4c4e-b35e-fa3703de95ed" volumeName="kubernetes.io/projected/15d998c6-eb92-4c4e-b35e-fa3703de95ed-kube-api-access-tjjw8" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029516    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029529    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029541    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029560    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ccb6ab-b665-4753-96a2-9ce1fa145666" volumeName="kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-tls" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029572    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb21c178-123e-4c5c-a0f3-e59e5aab8927" volumeName="kubernetes.io/projected/eb21c178-123e-4c5c-a0f3-e59e5aab8927-kube-api-access-65rvz" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029584    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029605    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029626    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029637    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029658    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/projected/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-kube-api-access-7kmsg" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029670    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40daba9a-1280-486d-b7f4-047d0efdab2e" volumeName="kubernetes.io/projected/40daba9a-1280-486d-b7f4-047d0efdab2e-kube-api-access-blgnm" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029682    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9225fde2-c7af-4b4e-80c0-6a473f655fec" volumeName="kubernetes.io/secret/9225fde2-c7af-4b4e-80c0-6a473f655fec-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029702    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9713304b-d13e-4e2c-b460-6a02c9adf5b4" volumeName="kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cni-binary-copy" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029714    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" volumeName="kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-binary-copy" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029727    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ccb6ab-b665-4753-96a2-9ce1fa145666" volumeName="kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029751    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/empty-dir/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-tmp" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029764    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0977c82f-e896-4817-b4bf-eb1b056060c6" volumeName="kubernetes.io/projected/0977c82f-e896-4817-b4bf-eb1b056060c6-kube-api-access-d8729" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029775    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029788    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9225fde2-c7af-4b4e-80c0-6a473f655fec" volumeName="kubernetes.io/projected/9225fde2-c7af-4b4e-80c0-6a473f655fec-kube-api-access-mbssx" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029802    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3196f084-b0f3-4145-be7f-0d2b63dd8019" volumeName="kubernetes.io/empty-dir/3196f084-b0f3-4145-be7f-0d2b63dd8019-tmp-dir" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029844    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029857    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029868    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029879    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029893    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15d998c6-eb92-4c4e-b35e-fa3703de95ed" volumeName="kubernetes.io/secret/15d998c6-eb92-4c4e-b35e-fa3703de95ed-cert" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029904    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb511bd-1e7a-44c1-b027-964d3fb57d08" volumeName="kubernetes.io/configmap/1eb511bd-1e7a-44c1-b027-964d3fb57d08-iptables-alerter-script" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029915    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb511bd-1e7a-44c1-b027-964d3fb57d08" volumeName="kubernetes.io/projected/1eb511bd-1e7a-44c1-b027-964d3fb57d08-kube-api-access-2dncp" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029928    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8edbb9a8-e48f-428f-9967-61dd8333133e" volumeName="kubernetes.io/projected/8edbb9a8-e48f-428f-9967-61dd8333133e-kube-api-access-qtkpv" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029940    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ccb6ab-b665-4753-96a2-9ce1fa145666" volumeName="kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-metrics-client-ca" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029950    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb21c178-123e-4c5c-a0f3-e59e5aab8927" volumeName="kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.029964    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/empty-dir/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-audit-log" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030001    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9713304b-d13e-4e2c-b460-6a02c9adf5b4" volumeName="kubernetes.io/projected/9713304b-d13e-4e2c-b460-6a02c9adf5b4-kube-api-access-f9ns6" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030014    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1c76fe7-8625-4a41-a039-af53c4978a94" volumeName="kubernetes.io/empty-dir/b1c76fe7-8625-4a41-a039-af53c4978a94-tmp-dir" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030027    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ccb6ab-b665-4753-96a2-9ce1fa145666" volumeName="kubernetes.io/projected/e3ccb6ab-b665-4753-96a2-9ce1fa145666-kube-api-access-wh7vl" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030038    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ccb6ab-b665-4753-96a2-9ce1fa145666" volumeName="kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030051    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/projected/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-kube-api-access-46t45" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030064    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030076    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e29be6-c48c-4bc8-a601-a0e337715b61" volumeName="kubernetes.io/configmap/d6e29be6-c48c-4bc8-a601-a0e337715b61-konnectivity-ca" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030087    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" volumeName="kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030107    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb21c178-123e-4c5c-a0f3-e59e5aab8927" volumeName="kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030119    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb21c178-123e-4c5c-a0f3-e59e5aab8927" volumeName="kubernetes.io/secret/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030130    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030143    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030155    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030167    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9713304b-d13e-4e2c-b460-6a02c9adf5b4" volumeName="kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-daemon-config" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030178    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030189    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0977c82f-e896-4817-b4bf-eb1b056060c6" volumeName="kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-tuned" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030201    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3196f084-b0f3-4145-be7f-0d2b63dd8019" volumeName="kubernetes.io/projected/3196f084-b0f3-4145-be7f-0d2b63dd8019-kube-api-access-7h9mw" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030215    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030246    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030259    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" volumeName="kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030272    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2d69636-c98b-4456-93be-168d230451b6" volumeName="kubernetes.io/configmap/f2d69636-c98b-4456-93be-168d230451b6-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030287    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030298    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030311    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bc6d4db-06d7-4767-bef1-d90cfb51dc88" volumeName="kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030323    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" volumeName="kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030340    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4fa7bf3-f448-451d-b46e-a885de36da57" volumeName="kubernetes.io/projected/a4fa7bf3-f448-451d-b46e-a885de36da57-kube-api-access-xc45c" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030352    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f983b9ee-1e60-4ec4-a196-e40be0d0ccd9" volumeName="kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030364    6481 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fcf9acbd-ada8-42c9-a659-32e906773a1f" volumeName="kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca" seLinuxMountContext=""
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030375    6481 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 14:12:02.030693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.030382    6481 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 14:12:02.043358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.043326    6481 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 14:12:02.059136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.059115    6481 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 14:12:02.059136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.059136    6481 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 14:12:02.059364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.059152    6481 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 14:12:02.059364 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.059192    6481 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 14:12:02.072288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.072260    6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="85d2fccf3b432d777b7d308140a31e3101b6c3845cdd1820c996e346488e865b" exitCode=0
Nov 03 14:12:02.072288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.072285    6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="435a9af9665e767c2b8856c60e02eef6ccd674b6a77379823d202339656558c7" exitCode=0
Nov 03 14:12:02.072288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.072293    6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="ef071166d03d533d40faeae6e780bb87746d8bfaed6585a9330744f305ffa324" exitCode=0
Nov 03 14:12:02.072484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.072300    6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="322564f5adbc888b1962a0f5a87ba85f4254e3b60a7a10189a9cb3de0c2316f6" exitCode=0
Nov 03 14:12:02.072484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.072305    6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="4b1b46811728d90140eba954d27b459e4f7965c4146f7c4c6ef8194129252d83" exitCode=0
Nov 03 14:12:02.072484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.072311    6481 generic.go:356] "Generic (PLEG): container finished" podID="d6e35132-a355-4ec6-a6f6-3ef83c5c2840" containerID="aee30b0ad07a679f8e797233c30758d9f92b710817f4b95d5e40e60654e59fbb" exitCode=0
Nov 03 14:12:02.075505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.075485    6481 generic.go:356] "Generic (PLEG): container finished" podID="86a96a710dae74d27a1c0a4680122307" containerID="f1755eccd356e1f2576c8b7b6e8d71487dffb5b8ba9d730ad3ccb4001cfe21ab" exitCode=0
Nov 03 14:12:02.076445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.076426    6481 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:12:02.076558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.076542    6481 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 14:12:02.083769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.083746    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4" exitCode=0
Nov 03 14:12:02.091712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.091691    6481 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:12:02.095657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.095636    6481 generic.go:356] "Generic (PLEG): container finished" podID="e3ccb6ab-b665-4753-96a2-9ce1fa145666" containerID="78834032a7ac6445acd4135cf0ad5c500a4e96699492d4c78de8450c9e18043c" exitCode=0
Nov 03 14:12:02.102425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.102403    6481 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:12:02.102558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.102528    6481 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 14:12:02.113562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.113541    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7" exitCode=0
Nov 03 14:12:02.133829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.133812    6481 manager.go:324] Recovery completed
Nov 03 14:12:02.156892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.156862    6481 cpu_manager.go:226] "Starting" policy="none"
Nov 03 14:12:02.156892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.156875    6481 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 14:12:02.156892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.156891    6481 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.156999    6481 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.157010    6481 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.157025    6481 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.157031    6481 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.157036    6481 policy_none.go:50] "Start"
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.157044    6481 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 14:12:02.157072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.157053    6481 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 14:12:02.158168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.158155    6481 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 14:12:02.158205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.158169    6481 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 14:12:02.158205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.158175    6481 policy_none.go:44] "Start"
Nov 03 14:12:02.159388 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.159372    6481 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 14:12:02.162649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.162621    6481 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 14:12:02.165605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.165589    6481 manager.go:341] "Starting Device Plugin manager"
Nov 03 14:12:02.166102 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.165618    6481 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 14:12:02.166180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166105    6481 server.go:87] "Starting device plugin registration server"
Nov 03 14:12:02.166414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166400    6481 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 14:12:02.166477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166414    6481 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 14:12:02.166526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166478    6481 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 14:12:02.166626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166579    6481 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 14:12:02.166726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166631    6481 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 14:12:02.166726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166638    6481 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 14:12:02.166825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.166688    6481 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T14:12:02.166614979Z","UUID":"0be4e849-f748-48ec-8e63-9973063de3f3","Handler":null,"Name":"","Endpoint":""}
Nov 03 14:12:02.168184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.168163    6481 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 14:12:02.168281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.168191    6481 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 14:12:02.168373 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.168350    6481 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 14:12:02.266532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.266502    6481 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 14:12:02.268673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.268653    6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 14:12:02.268772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.268685    6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 14:12:02.268772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.268697    6481 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-190.ec2.internal" event="NodeHasSufficientPID"
Nov 03 14:12:02.268772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.268736    6481 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.306708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.306684    6481 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.306815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.306747    6481 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.360196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360151    6481 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"]
Nov 03 14:12:02.360728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360692    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" event={"ID":"86a96a710dae74d27a1c0a4680122307","Type":"ContainerStarted","Data":"7124ae02151f24d10bf01a2423352d709846902cc6f189a8c85efed629fa9207"}
Nov 03 14:12:02.360811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360731    6481 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.360811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360735    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" event={"ID":"86a96a710dae74d27a1c0a4680122307","Type":"ContainerDied","Data":"f1755eccd356e1f2576c8b7b6e8d71487dffb5b8ba9d730ad3ccb4001cfe21ab"}
Nov 03 14:12:02.360811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360751    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal" event={"ID":"86a96a710dae74d27a1c0a4680122307","Type":"ContainerStarted","Data":"dbe7a8136bc2c8a4b7955d316b1ec5d212bf7360b109ff8413e34910322dcd6c"}
Nov 03 14:12:02.360811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360792    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal" event={"ID":"61a071342255b3898148bdccf890b4bc","Type":"ContainerStarted","Data":"43829e187da0f2adee276632f1d8d6eccfbef56d624cd04bc9dbd3459f8f0a5f"}
Nov 03 14:12:02.360811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360805    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal" event={"ID":"61a071342255b3898148bdccf890b4bc","Type":"ContainerStarted","Data":"096fca31bb6bc73500ee24ec6a0d4651d763e48419e12d54f35035529679e57b"}
Nov 03 14:12:02.361026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.360962    6481 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.389403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.389387    6481 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 14:12:02.389454 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.389424    6481 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.390880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.390868    6481 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 14:12:02.390925 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:02.390896    6481 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.478241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.478179    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.478241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.478203    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.478241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.478220    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61a071342255b3898148bdccf890b4bc-config\") pod \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" (UID: \"61a071342255b3898148bdccf890b4bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.578778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.578756    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.578837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.578782    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.578837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.578800    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61a071342255b3898148bdccf890b4bc-config\") pod \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" (UID: \"61a071342255b3898148bdccf890b4bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.578928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.578840    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/61a071342255b3898148bdccf890b4bc-config\") pod \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" (UID: \"61a071342255b3898148bdccf890b4bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.578928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.578857    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:02.578928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:02.578887    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86a96a710dae74d27a1c0a4680122307-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal\" (UID: \"86a96a710dae74d27a1c0a4680122307\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:03.011424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.011381    6481 apiserver.go:52] "Watching apiserver"
Nov 03 14:12:03.069211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.069160    6481 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 14:12:03.070196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.070165    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-190.ec2.internal","openshift-cluster-node-tuning-operator/tuned-k8qn5","openshift-dns/dns-default-fwrl7","openshift-multus/multus-9f6lw","openshift-ovn-kubernetes/ovnkube-node-f55sl","openshift-ingress-canary/ingress-canary-pkm2z","openshift-insights/insights-runtime-extractor-kws78","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-9vstd","openshift-network-operator/iptables-alerter-kmcrp","kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal","openshift-image-registry/node-ca-868d4","openshift-monitoring/node-exporter-n87hh","kube-system/global-pull-secret-syncer-8flw8","openshift-monitoring/alertmanager-main-0","kube-system/konnectivity-agent-7t76q","openshift-multus/multus-additional-cni-plugins-n6kkm","openshift-network-diagnostics/network-check-target-s7z9p","openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl","openshift-monitoring/thanos-querier-76f8554fdc-w8wsw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn","openshift-monitoring/metrics-server-6679c484c8-thcsr","openshift-dns/node-resolver-g9r7l"]
Nov 03 14:12:03.089774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089752    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.089774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089758    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-trbj6\""
Nov 03 14:12:03.089774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089761    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.090060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089943    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-95wl7\""
Nov 03 14:12:03.090060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089963    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.090060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089977    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 14:12:03.090060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089974    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.090060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.089995    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lqmd6\""
Nov 03 14:12:03.090060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.090010    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 14:12:03.090369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.090064    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 14:12:03.090369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.090151    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 14:12:03.090369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.090209    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rmj5t\""
Nov 03 14:12:03.090369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.090326    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qhrll\""
Nov 03 14:12:03.091246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091217    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 14:12:03.091363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091348    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 14:12:03.091446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091434    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.091528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091512    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.091589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091520    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.091638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091601    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.091672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091636    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 14:12:03.091672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091657    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 14:12:03.091750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091668    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2ft6v\""
Nov 03 14:12:03.091810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091795    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 14:12:03.091882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.091871    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qgxkz\""
Nov 03 14:12:03.100265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.100220    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-bgwlk\""
Nov 03 14:12:03.100265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.100221    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 14:12:03.100887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.100872    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 14:12:03.101003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.100991    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.101058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101048    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.101101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101087    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.101155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101143    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.101206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101193    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.101352 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101340    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 14:12:03.101387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101377    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 14:12:03.101419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101408    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.101447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101435    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.101492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101482    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 14:12:03.101552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.101540    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 14:12:03.102147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.102135    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.102259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.102248    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 14:12:03.102307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.102294    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 14:12:03.102432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.102422    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.102949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.102938    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q495v\""
Nov 03 14:12:03.103170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103156    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.103242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103209    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xm8pf\""
Nov 03 14:12:03.103242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103211    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghsgd\""
Nov 03 14:12:03.103345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103276    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tb44c\""
Nov 03 14:12:03.103345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103320    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hlrhs\""
Nov 03 14:12:03.103345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103335    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ml487\""
Nov 03 14:12:03.103345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103340    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-r6xhc\""
Nov 03 14:12:03.103491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103348    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 14:12:03.103491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103364    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 14:12:03.103491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103337    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.103491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103426    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.103491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103445    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 14:12:03.103777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103758    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 14:12:03.103946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.103925    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 14:12:03.104099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.104069    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 14:12:03.104274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.104257    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 14:12:03.104449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.104431    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 14:12:03.104509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.104471    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 14:12:03.104618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.104597    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 14:12:03.104672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.104634    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 14:12:03.105592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105569    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 14:12:03.105685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105608    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 14:12:03.105685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105624    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 14:12:03.105777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105690    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 14:12:03.105821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105800    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 14:12:03.105821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105805    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 14:12:03.105925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105909    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tpn6s\""
Nov 03 14:12:03.105979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105927    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 14:12:03.105979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105969    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 14:12:03.106068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.105981    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 14:12:03.106116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106069    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 14:12:03.106116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106089    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l8fm6\""
Nov 03 14:12:03.106210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106123    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 14:12:03.106283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106215    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 14:12:03.106330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106287    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nkgzv\""
Nov 03 14:12:03.106330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106312    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ndkr\""
Nov 03 14:12:03.106407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106357    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjlcb\""
Nov 03 14:12:03.106489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106468    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 14:12:03.106616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106499    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 14:12:03.106679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106613    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 14:12:03.106679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.106631    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f799n8dkgte3s\""
Nov 03 14:12:03.107085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.107057    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 14:12:03.107590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.107520    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 14:12:03.119237 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.119200    6481 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:03.129326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.129310    6481 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 14:12:03.142344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.142324    6481 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 14:12:03.142434 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:03.142367    6481 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-190.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-190.ec2.internal"
Nov 03 14:12:03.181377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181356    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.181450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181381    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.181450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181398    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.181450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181414    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.181450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181429    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.181450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181446    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-multus\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.181698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181463    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.181698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181486    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-env-overrides\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.181698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181538    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-modprobe-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.181698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181619    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-tuned\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.181698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181663    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.181698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181663    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-hostroot\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181698    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181708    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-tuned\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181698    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181705    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181755    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-socket-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181699    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-env-overrides\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181784    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181847    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2d69636-c98b-4456-93be-168d230451b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181867    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9927b257-7027-4ef3-aa19-bde73b4d93b9-original-pull-secret\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181884    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-dbus\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181888    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181904    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181921    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-conf\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.181935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181941    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-daemon-config\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181957    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjw8\" (UniqueName: \"kubernetes.io/projected/15d998c6-eb92-4c4e-b35e-fa3703de95ed-kube-api-access-tjjw8\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.181993    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f2d69636-c98b-4456-93be-168d230451b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182106    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-bin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182113    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-daemon-config\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182141    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182186    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2d69636-c98b-4456-93be-168d230451b6-crio-socket\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182213    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxbk\" (UniqueName: \"kubernetes.io/projected/f2d69636-c98b-4456-93be-168d230451b6-kube-api-access-9xxbk\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182266    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182287    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182302    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cnibin\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182319    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-systemd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182372    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-var-lib-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182380    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182417    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182434    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182441    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2d69636-c98b-4456-93be-168d230451b6-data-volume\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182487    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f2d69636-c98b-4456-93be-168d230451b6-data-volume\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182487    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-client-certs\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.182506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182515    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-multus-certs\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182533    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-etc-kubernetes\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182557    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182573    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-slash\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182586    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-etc-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182690    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-audit-log\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182715    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8729\" (UniqueName: \"kubernetes.io/projected/0977c82f-e896-4817-b4bf-eb1b056060c6-kube-api-access-d8729\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182731    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182767    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182804    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-registration-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182832    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182836    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-audit-log\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182831    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3196f084-b0f3-4145-be7f-0d2b63dd8019-config-volume\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182873    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-root\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182908    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9225fde2-c7af-4b4e-80c0-6a473f655fec-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182916    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182952    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29be6-c48c-4bc8-a601-a0e337715b61-konnectivity-ca\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182978    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.182992    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-sys\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183014    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3196f084-b0f3-4145-be7f-0d2b63dd8019-config-volume\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183033    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-tls\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183051    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cnibin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183067    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183106    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9225fde2-c7af-4b4e-80c0-6a473f655fec-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183133    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-netns\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183160    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-socket-dir-parent\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183174    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29be6-c48c-4bc8-a601-a0e337715b61-konnectivity-ca\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183185    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183203    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e29be6-c48c-4bc8-a601-a0e337715b61-agent-certs\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:12:03.183286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183252    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-tls\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183333    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1c76fe7-8625-4a41-a039-af53c4978a94-hosts-file\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183370    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr6tm\" (UniqueName: \"kubernetes.io/projected/b1c76fe7-8625-4a41-a039-af53c4978a94-kube-api-access-fr6tm\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183396    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-tmp\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183406    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d6e29be6-c48c-4bc8-a601-a0e337715b61-agent-certs\") pod \"konnectivity-agent-7t76q\" (UID: \"d6e29be6-c48c-4bc8-a601-a0e337715b61\") " pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183419    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-tmp\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183442    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183504    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-tmp\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183524    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0977c82f-e896-4817-b4bf-eb1b056060c6-tmp\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183562    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183590    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbssx\" (UniqueName: \"kubernetes.io/projected/9225fde2-c7af-4b4e-80c0-6a473f655fec-kube-api-access-mbssx\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183637    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183663    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-log-socket\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183708    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-bin\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183821    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-tls\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183853    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8edbb9a8-e48f-428f-9967-61dd8333133e-serviceca\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183878    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.183905    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184035    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184063    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-kubelet\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184097    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-metrics-server-audit-profiles\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184121    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-host\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184145    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-netd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184172    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184265    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d998c6-eb92-4c4e-b35e-fa3703de95ed-cert\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184293    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb511bd-1e7a-44c1-b027-964d3fb57d08-host-slash\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184318    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-lib-modules\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184342    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-script-lib\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184368    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184393    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1eb511bd-1e7a-44c1-b027-964d3fb57d08-iptables-alerter-script\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184421    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184447    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-client-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184472    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysconfig\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184502    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184549    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.184623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184576    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184890    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc45c\" (UniqueName: \"kubernetes.io/projected/a4fa7bf3-f448-451d-b46e-a885de36da57-kube-api-access-xc45c\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184922    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184951    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpfvh\" (UniqueName: \"kubernetes.io/projected/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-kube-api-access-mpfvh\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.184995    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c76fe7-8625-4a41-a039-af53c4978a94-tmp-dir\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185031    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-ovn\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185056    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185081    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-sys\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185108    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-textfile\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185146    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cni-binary-copy\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185175    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185204    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-node-log\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185253    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185281    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-sys-fs\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185306    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185456    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1eb511bd-1e7a-44c1-b027-964d3fb57d08-iptables-alerter-script\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185461    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185497    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185520    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185618    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185645    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185651    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-tls\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185682    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185654    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185700    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185718    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-textfile\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185653    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-script-lib\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185731    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-kubelet\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185719    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1c76fe7-8625-4a41-a039-af53c4978a94-tmp-dir\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185765    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185865    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185879    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cni-binary-copy\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185909    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-system-cni-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185911    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185923    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15d998c6-eb92-4c4e-b35e-fa3703de95ed-cert\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185955    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ns6\" (UniqueName: \"kubernetes.io/projected/9713304b-d13e-4e2c-b460-6a02c9adf5b4-kube-api-access-f9ns6\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185964    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185985    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.185995    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-config\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186063    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-metrics-server-audit-profiles\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186113    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-systemd\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186142    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-var-lib-kubelet\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186158    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8edbb9a8-e48f-428f-9967-61dd8333133e-serviceca\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.186191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186176    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-wtmp\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186244    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovnkube-config\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186264    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186291    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186344    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186382    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovn-node-metrics-cert\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186466    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3196f084-b0f3-4145-be7f-0d2b63dd8019-tmp-dir\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186508    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186544    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-kubernetes\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186569    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-run\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186568    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3196f084-b0f3-4145-be7f-0d2b63dd8019-tmp-dir\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186589    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb21c178-123e-4c5c-a0f3-e59e5aab8927-ovn-node-metrics-cert\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186589    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186674    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186707    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-conf-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186741    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40daba9a-1280-486d-b7f4-047d0efdab2e-metrics-certs\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186757    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186787    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dncp\" (UniqueName: \"kubernetes.io/projected/1eb511bd-1e7a-44c1-b027-964d3fb57d08-kube-api-access-2dncp\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186844    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-systemd-units\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186859    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186934    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmsg\" (UniqueName: \"kubernetes.io/projected/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-kube-api-access-7kmsg\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.186964    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkpv\" (UniqueName: \"kubernetes.io/projected/8edbb9a8-e48f-428f-9967-61dd8333133e-kube-api-access-qtkpv\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187008    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-accelerators-collector-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187104    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9mw\" (UniqueName: \"kubernetes.io/projected/3196f084-b0f3-4145-be7f-0d2b63dd8019-kube-api-access-7h9mw\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187130    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187265    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgnm\" (UniqueName: \"kubernetes.io/projected/40daba9a-1280-486d-b7f4-047d0efdab2e-kube-api-access-blgnm\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187304    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7vl\" (UniqueName: \"kubernetes.io/projected/e3ccb6ab-b665-4753-96a2-9ce1fa145666-kube-api-access-wh7vl\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187308    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187388    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-accelerators-collector-config\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187406    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-device-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187503    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187556    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2d69636-c98b-4456-93be-168d230451b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187625    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-metrics-client-ca\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187645    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187662    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187683    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-metrics-client-ca\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187708    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t45\" (UniqueName: \"kubernetes.io/projected/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-kube-api-access-46t45\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187743    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-system-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187761    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-k8s-cni-cncf-io\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187835    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187847    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f2d69636-c98b-4456-93be-168d230451b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187887    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3196f084-b0f3-4145-be7f-0d2b63dd8019-metrics-tls\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187895    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3ccb6ab-b665-4753-96a2-9ce1fa145666-metrics-client-ca\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187911    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-metrics-client-ca\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.187903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187920    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-kubelet-config\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187952    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187959    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.187988    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65rvz\" (UniqueName: \"kubernetes.io/projected/eb21c178-123e-4c5c-a0f3-e59e5aab8927-kube-api-access-65rvz\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188050    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cni-binary-copy\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188055    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188113    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3196f084-b0f3-4145-be7f-0d2b63dd8019-metrics-tls\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188133    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188150    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-netns\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188166    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188178    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188182    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188203    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-grpc-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188246    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188316    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edbb9a8-e48f-428f-9967-61dd8333133e-host\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188414    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.189443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.188471    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-secret-grpc-tls\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.190644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190626    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 14:12:03.190644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190640    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 14:12:03.190763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190640    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190827    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-21aqngq8nfakl\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190857    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190863    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190861    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190878    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190862    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190860    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190882    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 14:12:03.190937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190913    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9g39dtaijeu5d\""
Nov 03 14:12:03.191346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.190971    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 14:12:03.191346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.191029    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 14:12:03.191346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.191164    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwv7d\""
Nov 03 14:12:03.192029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.192012    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.192640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.192621    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 14:12:03.192747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.192656    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 14:12:03.192798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.192756    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-secret-metrics-server-client-certs\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.192798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.192784    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.192857 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.192843    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.194289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.194271    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.194381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.194332    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.194423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.194392    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.195570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.195553    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.195634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.195603    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.195686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.195672    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.196691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.196676    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.196767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.196746    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-client-ca-bundle\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.197965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.197943    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.199123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.199096    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.199219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.199203    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.200145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.200127    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.205442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205209    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr6tm\" (UniqueName: \"kubernetes.io/projected/b1c76fe7-8625-4a41-a039-af53c4978a94-kube-api-access-fr6tm\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.205442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205273    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxbk\" (UniqueName: \"kubernetes.io/projected/f2d69636-c98b-4456-93be-168d230451b6-kube-api-access-9xxbk\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.205442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205334    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8729\" (UniqueName: \"kubernetes.io/projected/0977c82f-e896-4817-b4bf-eb1b056060c6-kube-api-access-d8729\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.205614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205439    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjw8\" (UniqueName: \"kubernetes.io/projected/15d998c6-eb92-4c4e-b35e-fa3703de95ed-kube-api-access-tjjw8\") pod \"ingress-canary-pkm2z\" (UID: \"15d998c6-eb92-4c4e-b35e-fa3703de95ed\") " pod="openshift-ingress-canary/ingress-canary-pkm2z"
Nov 03 14:12:03.206003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205973    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") pod \"alertmanager-main-0\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:03.206067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205985    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") pod \"prometheus-k8s-0\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:03.206067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.205990    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbssx\" (UniqueName: \"kubernetes.io/projected/9225fde2-c7af-4b4e-80c0-6a473f655fec-kube-api-access-mbssx\") pod \"monitoring-plugin-77f4f76594-c2pfl\" (UID: \"9225fde2-c7af-4b4e-80c0-6a473f655fec\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:12:03.206184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.206165    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpfvh\" (UniqueName: \"kubernetes.io/projected/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-kube-api-access-mpfvh\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.206321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.206300    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc45c\" (UniqueName: \"kubernetes.io/projected/a4fa7bf3-f448-451d-b46e-a885de36da57-kube-api-access-xc45c\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.206819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.206798    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dncp\" (UniqueName: \"kubernetes.io/projected/1eb511bd-1e7a-44c1-b027-964d3fb57d08-kube-api-access-2dncp\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.207000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.206951    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ns6\" (UniqueName: \"kubernetes.io/projected/9713304b-d13e-4e2c-b460-6a02c9adf5b4-kube-api-access-f9ns6\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.208648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.208622    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9mw\" (UniqueName: \"kubernetes.io/projected/3196f084-b0f3-4145-be7f-0d2b63dd8019-kube-api-access-7h9mw\") pod \"dns-default-fwrl7\" (UID: \"3196f084-b0f3-4145-be7f-0d2b63dd8019\") " pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:03.208748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.208671    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgnm\" (UniqueName: \"kubernetes.io/projected/40daba9a-1280-486d-b7f4-047d0efdab2e-kube-api-access-blgnm\") pod \"network-metrics-daemon-9vstd\" (UID: \"40daba9a-1280-486d-b7f4-047d0efdab2e\") " pod="openshift-multus/network-metrics-daemon-9vstd"
Nov 03 14:12:03.208748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.208689    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmsg\" (UniqueName: \"kubernetes.io/projected/0bc6d4db-06d7-4767-bef1-d90cfb51dc88-kube-api-access-7kmsg\") pod \"metrics-server-6679c484c8-thcsr\" (UID: \"0bc6d4db-06d7-4767-bef1-d90cfb51dc88\") " pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:03.209170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.209149    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7vl\" (UniqueName: \"kubernetes.io/projected/e3ccb6ab-b665-4753-96a2-9ce1fa145666-kube-api-access-wh7vl\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.209496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.209474    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65rvz\" (UniqueName: \"kubernetes.io/projected/eb21c178-123e-4c5c-a0f3-e59e5aab8927-kube-api-access-65rvz\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.209566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.209503    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t45\" (UniqueName: \"kubernetes.io/projected/f983b9ee-1e60-4ec4-a196-e40be0d0ccd9-kube-api-access-46t45\") pod \"thanos-querier-76f8554fdc-w8wsw\" (UID: \"f983b9ee-1e60-4ec4-a196-e40be0d0ccd9\") " pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:03.209718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.209697    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkpv\" (UniqueName: \"kubernetes.io/projected/8edbb9a8-e48f-428f-9967-61dd8333133e-kube-api-access-qtkpv\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.210190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.210157    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlms\" (UniqueName: \"kubernetes.io/projected/167d067e-3824-4add-b57b-9ba2bfc6c200-kube-api-access-7zlms\") pod \"network-check-target-s7z9p\" (UID: \"167d067e-3824-4add-b57b-9ba2bfc6c200\") " pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:12:03.288991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.288924    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-ovn\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.288991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.288953    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-sys\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.288991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.288970    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-node-log\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.288991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.288985    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-sys-fs\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.288999    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289050    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-node-log\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289066    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-sys\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289054    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-sys-fs\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289077    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-kubelet\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289117    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-system-cni-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289147    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-systemd\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289155    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-system-cni-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289158    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-kubelet\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289064    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-ovn\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289177    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-var-lib-kubelet\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289181    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289204    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-wtmp\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289242    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-var-lib-kubelet\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289248    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289250    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-systemd\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.289284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289277    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-kubernetes\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289299    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-run\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289305    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-kubernetes\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289279    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289324    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-conf-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289330    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-node-exporter-wtmp\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289340    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-run\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289348    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-systemd-units\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289364    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-conf-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289376    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-device-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289391    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-systemd-units\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289403    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-system-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289423    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-device-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289436    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-k8s-cni-cncf-io\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289453    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-system-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289460    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-kubelet-config\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289493    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-netns\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289495    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-k8s-cni-cncf-io\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289503    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-kubelet-config\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289518    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289535    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-netns\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289541    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edbb9a8-e48f-428f-9967-61dd8333133e-host\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289580    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289598    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8edbb9a8-e48f-428f-9967-61dd8333133e-host\") pod \"node-ca-868d4\" (UID: \"8edbb9a8-e48f-428f-9967-61dd8333133e\") " pod="openshift-image-registry/node-ca-868d4"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289601    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-multus\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289619    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-multus\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289640    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-modprobe-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289664    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-hostroot\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289688    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-socket-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289714    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289734    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-hostroot\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289738    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-dbus\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289763    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-conf\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289765    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-socket-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289781    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-modprobe-d\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289788    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-bin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289783    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289798    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9927b257-7027-4ef3-aa19-bde73b4d93b9-dbus\") pod \"global-pull-secret-syncer-8flw8\" (UID: \"9927b257-7027-4ef3-aa19-bde73b4d93b9\") " pod="kube-system/global-pull-secret-syncer-8flw8"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289812    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2d69636-c98b-4456-93be-168d230451b6-crio-socket\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289829    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-var-lib-cni-bin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289855    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cnibin\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289876    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysctl-conf\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289901    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-systemd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289874    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f2d69636-c98b-4456-93be-168d230451b6-crio-socket\") pod \"insights-runtime-extractor-kws78\" (UID: \"f2d69636-c98b-4456-93be-168d230451b6\") " pod="openshift-insights/insights-runtime-extractor-kws78"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289920    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6e35132-a355-4ec6-a6f6-3ef83c5c2840-cnibin\") pod \"multus-additional-cni-plugins-n6kkm\" (UID: \"d6e35132-a355-4ec6-a6f6-3ef83c5c2840\") " pod="openshift-multus/multus-additional-cni-plugins-n6kkm"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289925    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-var-lib-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289954    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-var-lib-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289953    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289979    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289959    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-run-systemd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.289991    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-multus-certs\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290011    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-etc-kubernetes\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290026    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-slash\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290043    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-etc-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290070    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-registration-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290072    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-slash\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290095    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-root\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290104    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-host-run-multus-certs\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290072    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-etc-kubernetes\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290122    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-etc-openvswitch\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290148    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-root\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.290109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290162    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290127    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-registration-dir\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290123    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-cni-dir\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290203    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-sys\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290253    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cnibin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290268    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e3ccb6ab-b665-4753-96a2-9ce1fa145666-sys\") pod \"node-exporter-n87hh\" (UID: \"e3ccb6ab-b665-4753-96a2-9ce1fa145666\") " pod="openshift-monitoring/node-exporter-n87hh"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290278    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-netns\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290301    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-socket-dir-parent\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290310    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-cnibin\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290339    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-run-netns\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290328    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1c76fe7-8625-4a41-a039-af53c4978a94-hosts-file\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290378    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1c76fe7-8625-4a41-a039-af53c4978a94-hosts-file\") pod \"node-resolver-g9r7l\" (UID: \"b1c76fe7-8625-4a41-a039-af53c4978a94\") " pod="openshift-dns/node-resolver-g9r7l"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290382    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-log-socket\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290383    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9713304b-d13e-4e2c-b460-6a02c9adf5b4-multus-socket-dir-parent\") pod \"multus-9f6lw\" (UID: \"9713304b-d13e-4e2c-b460-6a02c9adf5b4\") " pod="openshift-multus/multus-9f6lw"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290404    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-log-socket\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290405    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-bin\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290432    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-kubelet\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290454    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-host\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290464    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-bin\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290477    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-netd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290501    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-host\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290504    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb511bd-1e7a-44c1-b027-964d3fb57d08-host-slash\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290504    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-kubelet\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290532    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1eb511bd-1e7a-44c1-b027-964d3fb57d08-host-slash\") pod \"iptables-alerter-kmcrp\" (UID: \"1eb511bd-1e7a-44c1-b027-964d3fb57d08\") " pod="openshift-network-operator/iptables-alerter-kmcrp"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290540    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-cni-netd\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290549    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-lib-modules\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290566    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290581    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290598    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysconfig\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290629    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb21c178-123e-4c5c-a0f3-e59e5aab8927-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f55sl\" (UID: \"eb21c178-123e-4c5c-a0f3-e59e5aab8927\") " pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290641    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4fa7bf3-f448-451d-b46e-a885de36da57-etc-selinux\") pod \"aws-ebs-csi-driver-node-8dvjn\" (UID: \"a4fa7bf3-f448-451d-b46e-a885de36da57\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8dvjn"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290650    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-etc-sysconfig\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:03.291962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:03.290707    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0977c82f-e896-4817-b4bf-eb1b056060c6-lib-modules\") pod \"tuned-k8qn5\" (UID: \"0977c82f-e896-4817-b4bf-eb1b056060c6\") " pod="openshift-cluster-node-tuning-operator/tuned-k8qn5"
Nov 03 14:12:04.437323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:04.437292    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:04.656325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:04.656291    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:04.848847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:04.848776    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:12:04.857947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:04.857927    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fwrl7"
Nov 03 14:12:04.936875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:04.936851    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-76f8554fdc-w8wsw"
Nov 03 14:12:05.050803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:05.050768    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-c2pfl"
Nov 03 14:12:07.723063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:07.723034    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:08.004539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:08.004464    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:08.049118 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:08.049087    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:08.283819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:08.283746    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:09.198019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:09.197993    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6679c484c8-thcsr"
Nov 03 14:12:10.833012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:10.832973    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:10.833399 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:10.833171    6481 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 14:12:15.143974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:15.143892    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:15.368128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:15.368091    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:17.925627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:17.925591    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:18.149084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:18.149056    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:18.483551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:18.483523    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:18.704709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:18.704684    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:19.355355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:19.355320    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:20.351637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:20.351598    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:20.572407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:20.572379    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:21.226289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:21.226263    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7t76q"
Nov 03 14:12:21.345955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:21.345923    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:21.567963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:21.567894    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:22.945627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.945596    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:12:22.946076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.946022    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="alertmanager" containerID="cri-o://622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0" gracePeriod=120
Nov 03 14:12:22.946219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.946050    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-metric" containerID="cri-o://5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed" gracePeriod=120
Nov 03 14:12:22.946219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.946088    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="prom-label-proxy" containerID="cri-o://06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718" gracePeriod=120
Nov 03 14:12:22.946219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.946112    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy" containerID="cri-o://dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4" gracePeriod=120
Nov 03 14:12:22.946219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.946153    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="config-reloader" containerID="cri-o://43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda" gracePeriod=120
Nov 03 14:12:22.946219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:22.946099    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-web" containerID="cri-o://7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b" gracePeriod=120
Nov 03 14:12:23.194778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194746    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718" exitCode=0
Nov 03 14:12:23.194778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194774    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed" exitCode=0
Nov 03 14:12:23.194778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194783    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4" exitCode=0
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194795    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda" exitCode=0
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194806    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0" exitCode=0
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194820    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718"}
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194881    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed"}
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194908    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4"}
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194931    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda"}
Nov 03 14:12:23.194991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.194953    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0"}
Nov 03 14:12:23.454330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.454290    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:23.672621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:23.672580    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:24.181001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.180977    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.205488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.205456    6481 generic.go:356] "Generic (PLEG): container finished" podID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerID="7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b" exitCode=0
Nov 03 14:12:24.205599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.205537    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.205649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.205532    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b"}
Nov 03 14:12:24.205693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.205681    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fcf9acbd-ada8-42c9-a659-32e906773a1f","Type":"ContainerDied","Data":"457b7000c7de7bcc8088ce031787ca58dac96b7a22914b871a20895817d1f971"}
Nov 03 14:12:24.205733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.205715    6481 scope.go:122] "RemoveContainer" containerID="06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718"
Nov 03 14:12:24.213001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.212984    6481 scope.go:122] "RemoveContainer" containerID="5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed"
Nov 03 14:12:24.219590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.219574    6481 scope.go:122] "RemoveContainer" containerID="dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4"
Nov 03 14:12:24.225725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.225709    6481 scope.go:122] "RemoveContainer" containerID="7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b"
Nov 03 14:12:24.231939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.231919    6481 scope.go:122] "RemoveContainer" containerID="43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda"
Nov 03 14:12:24.238191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.238177    6481 scope.go:122] "RemoveContainer" containerID="622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0"
Nov 03 14:12:24.244533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.244518    6481 scope.go:122] "RemoveContainer" containerID="9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7"
Nov 03 14:12:24.250731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.250715    6481 scope.go:122] "RemoveContainer" containerID="06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718"
Nov 03 14:12:24.250987 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.250971    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718\": container with ID starting with 06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718 not found: ID does not exist" containerID="06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718"
Nov 03 14:12:24.251035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.250994    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718"} err="failed to get container status \"06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718\": rpc error: code = NotFound desc = could not find container \"06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718\": container with ID starting with 06164b5538fe3ba3eea2258cbaee241dbb5c5907afb7ede43df650b5b5987718 not found: ID does not exist"
Nov 03 14:12:24.251035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251020    6481 scope.go:122] "RemoveContainer" containerID="5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed"
Nov 03 14:12:24.251270 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.251255    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed\": container with ID starting with 5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed not found: ID does not exist" containerID="5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed"
Nov 03 14:12:24.251334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251275    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed"} err="failed to get container status \"5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed\": rpc error: code = NotFound desc = could not find container \"5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed\": container with ID starting with 5e7dad894caef9517d339daee53ee0df54058ed1e285af61fd0d1b45fb946eed not found: ID does not exist"
Nov 03 14:12:24.251334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251286    6481 scope.go:122] "RemoveContainer" containerID="dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4"
Nov 03 14:12:24.251532 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.251513    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4\": container with ID starting with dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4 not found: ID does not exist" containerID="dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4"
Nov 03 14:12:24.251576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251537    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4"} err="failed to get container status \"dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4\": rpc error: code = NotFound desc = could not find container \"dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4\": container with ID starting with dda5a9785ec29bf36a64adeb903a892e845a1ac4c29ac6b12ceca07bc76915f4 not found: ID does not exist"
Nov 03 14:12:24.251576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251552    6481 scope.go:122] "RemoveContainer" containerID="7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b"
Nov 03 14:12:24.251769 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.251755    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b\": container with ID starting with 7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b not found: ID does not exist" containerID="7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b"
Nov 03 14:12:24.251809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251772    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b"} err="failed to get container status \"7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b\": rpc error: code = NotFound desc = could not find container \"7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b\": container with ID starting with 7a3ecf7ed72379a3f8abdc9cd9622c6e87e2d43862e1460c52dc64cef8b4826b not found: ID does not exist"
Nov 03 14:12:24.251809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.251792    6481 scope.go:122] "RemoveContainer" containerID="43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda"
Nov 03 14:12:24.251997 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.251982    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda\": container with ID starting with 43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda not found: ID does not exist" containerID="43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda"
Nov 03 14:12:24.252037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.252000    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda"} err="failed to get container status \"43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda\": rpc error: code = NotFound desc = could not find container \"43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda\": container with ID starting with 43d0a1cba527ecf3503738f6acb1fef9cb3a37a6191e4d3a706360fdc49cafda not found: ID does not exist"
Nov 03 14:12:24.252037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.252012    6481 scope.go:122] "RemoveContainer" containerID="622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0"
Nov 03 14:12:24.252713 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.252184    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0\": container with ID starting with 622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0 not found: ID does not exist" containerID="622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0"
Nov 03 14:12:24.252713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.252200    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0"} err="failed to get container status \"622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0\": rpc error: code = NotFound desc = could not find container \"622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0\": container with ID starting with 622e337e9182bcee07e8273cd9ebf0c20002b650cb8e89fdf4681d845fd742e0 not found: ID does not exist"
Nov 03 14:12:24.252713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.252211    6481 scope.go:122] "RemoveContainer" containerID="9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7"
Nov 03 14:12:24.252713 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:24.252395    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7\": container with ID starting with 9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7 not found: ID does not exist" containerID="9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7"
Nov 03 14:12:24.252713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.252420    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7"} err="failed to get container status \"9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7\": rpc error: code = NotFound desc = could not find container \"9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7\": container with ID starting with 9ea3b6474969560dd762583af5964195d832ce378eaecf9c615b9c2ae9d762d7 not found: ID does not exist"
Nov 03 14:12:24.257707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257690    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.257774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257723    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.257774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257759    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.257850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257782    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.257850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257805    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.257850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257833    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257861    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257890    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.257939    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258008    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258037    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258065    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258100    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") pod \"fcf9acbd-ada8-42c9-a659-32e906773a1f\" (UID: \"fcf9acbd-ada8-42c9-a659-32e906773a1f\") "
Nov 03 14:12:24.258167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258128    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:12:24.258435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258197    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:24.258435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258345    6481 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-main-db\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.258435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.258366    6481 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-metrics-client-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.259119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.259034    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:24.261499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.261327    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:12:24.261695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.261395    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.261695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.261589    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.261838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.261778    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "kube-api-access-67zb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:24.261838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.261792    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.262980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.262958    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.263408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.263379    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:24.263582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.263559    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.265960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.265938    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.272831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.272665    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config" pod "fcf9acbd-ada8-42c9-a659-32e906773a1f" (UID: "fcf9acbd-ada8-42c9-a659-32e906773a1f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:24.358796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358770    6481 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.358796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358800    6481 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358840    6481 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-web-config\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358853    6481 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-tls-assets\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358866    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zb7\" (UniqueName: \"kubernetes.io/projected/fcf9acbd-ada8-42c9-a659-32e906773a1f-kube-api-access-67zb7\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358878    6481 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-cluster-tls-config\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358890    6481 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-main-tls\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358903    6481 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fcf9acbd-ada8-42c9-a659-32e906773a1f-config-out\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358917    6481 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358930    6481 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcf9acbd-ada8-42c9-a659-32e906773a1f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.358943    6481 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fcf9acbd-ada8-42c9-a659-32e906773a1f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:24.554885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.554825    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:12:24.568062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.568040    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:12:24.597665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.597639    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:12:24.597948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.597930    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-web"
Nov 03 14:12:24.598029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.597949    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-web"
Nov 03 14:12:24.598029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.597982    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="config-reloader"
Nov 03 14:12:24.598029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.597990    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="config-reloader"
Nov 03 14:12:24.598029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598008    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-metric"
Nov 03 14:12:24.598029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598016    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-metric"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598031    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="prom-label-proxy"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598038    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="prom-label-proxy"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598050    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="init-config-reloader"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598058    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="init-config-reloader"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598069    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="alertmanager"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598076    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="alertmanager"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598091    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598098    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598163    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="alertmanager"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598175    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598184    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-web"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598197    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="config-reloader"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598209    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="kube-rbac-proxy-metric"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598219    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="prom-label-proxy"
Nov 03 14:12:24.598298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.598246    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" containerName="init-config-reloader"
Nov 03 14:12:24.603564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.603544    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.620929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.620906    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 14:12:24.620929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.620908    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 14:12:24.621053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.620977    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 14:12:24.621053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.620979    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 14:12:24.621053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.620990    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 14:12:24.621222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.621208    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 14:12:24.621288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.621274    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 14:12:24.621324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.621277    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 14:12:24.621358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.621330    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2ndkr\""
Nov 03 14:12:24.634764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.634745    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 14:12:24.635735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.635719    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:12:24.662266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662243    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zb8n\" (UniqueName: \"kubernetes.io/projected/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-kube-api-access-7zb8n\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662271    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662306    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662380    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662404    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662445    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662513    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-web-config\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662572    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662595    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662619    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662640    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-config-out\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662683    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.662771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.662755    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763285    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763315    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763331    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763353    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-web-config\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763373    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763387    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763416    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763432    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-config-out\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763446    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763472    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763502    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zb8n\" (UniqueName: \"kubernetes.io/projected/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-kube-api-access-7zb8n\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763530    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.763697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763557    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.764009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.763693    6481 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 14:12:24.764532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.764190    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.764532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.764355    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.764688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.764552    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766394    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-config-out\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766534    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766591    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766738    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766823    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766877    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-web-config\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.766966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.766924    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.767293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.767272    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.768322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.768305    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.785385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.785368    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zb8n\" (UniqueName: \"kubernetes.io/projected/2d49b0b8-b06b-4a64-8f1d-b1f0a6951749-kube-api-access-7zb8n\") pod \"alertmanager-main-0\" (UID: \"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:24.927139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:24.927117    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 14:12:25.197435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:25.197369    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:25.250955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:25.250927    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 14:12:25.253814 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:25.253779    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d49b0b8_b06b_4a64_8f1d_b1f0a6951749.slice/crio-cec826cb8cf2a1f2ed05373746fbfdfba47bff3505b52b50e62cab4478bfd1a7 WatchSource:0}: Error finding container cec826cb8cf2a1f2ed05373746fbfdfba47bff3505b52b50e62cab4478bfd1a7: Status 404 returned error can't find the container with id cec826cb8cf2a1f2ed05373746fbfdfba47bff3505b52b50e62cab4478bfd1a7
Nov 03 14:12:25.420647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:25.420620    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:26.064271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:26.064224    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf9acbd-ada8-42c9-a659-32e906773a1f" path="/var/lib/kubelet/pods/fcf9acbd-ada8-42c9-a659-32e906773a1f/volumes"
Nov 03 14:12:26.218287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:26.218253    6481 generic.go:356] "Generic (PLEG): container finished" podID="2d49b0b8-b06b-4a64-8f1d-b1f0a6951749" containerID="392625919bbfd9fb3a8fe719d23de5a2f65926a464ccba3741526a08bd1edc56" exitCode=0
Nov 03 14:12:26.218670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:26.218317    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerDied","Data":"392625919bbfd9fb3a8fe719d23de5a2f65926a464ccba3741526a08bd1edc56"}
Nov 03 14:12:26.218670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:26.218391    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"cec826cb8cf2a1f2ed05373746fbfdfba47bff3505b52b50e62cab4478bfd1a7"}
Nov 03 14:12:26.356470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:26.356446    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f55sl"
Nov 03 14:12:27.225520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.225488    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"349d00cf2231d4d7d5f5870e2b116a1fc8442b10aabeb6166f9198388749c7af"}
Nov 03 14:12:27.225893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.225549    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"15528940fa50f0f4e41d976864e428753d52585a8de4bda85936eca0557f75f3"}
Nov 03 14:12:27.225893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.225574    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"d4630f4ba08a946b2fc256188dc06f29fcf5c322ed3366f8a19c4efcc6b6ed97"}
Nov 03 14:12:27.225893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.225599    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"25142176934b9317c3f500099cb9ce1ab5f9242f839de5693ac1eb6948a3ceef"}
Nov 03 14:12:27.225893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.225621    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"f0a8eab91d847b5e622075a84b2cc66dd05f33ff039fe98761d48c1aede7b6d4"}
Nov 03 14:12:27.225893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.225643    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d49b0b8-b06b-4a64-8f1d-b1f0a6951749","Type":"ContainerStarted","Data":"37f6d4e393a7b4f450f90bccabd3ec0e3dc8730638b1729822ed2def15b867a5"}
Nov 03 14:12:27.320168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.320112    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.309266073 podStartE2EDuration="3.320098774s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.832701ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:27.319654402 +0000 UTC m=+25.369685588" watchObservedRunningTime="2026-11-03 14:12:27.320098774 +0000 UTC m=+25.370129950"
Nov 03 14:12:27.553218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553142    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:12:27.553485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553458    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy" containerID="cri-o://4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859" gracePeriod=600
Nov 03 14:12:27.553578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553505    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="config-reloader" containerID="cri-o://9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7" gracePeriod=600
Nov 03 14:12:27.553578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553524    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-web" containerID="cri-o://d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41" gracePeriod=600
Nov 03 14:12:27.553578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553516    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="thanos-sidecar" containerID="cri-o://07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757" gracePeriod=600
Nov 03 14:12:27.553578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553454    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="prometheus" containerID="cri-o://64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116" gracePeriod=600
Nov 03 14:12:27.553745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.553512    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf" gracePeriod=600
Nov 03 14:12:27.624358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.624327    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:27.708662 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:27.708608    6481 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 14:12:27.708825 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:27.708682    6481 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 14:12:27.708825 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:27.708701    6481 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 03 14:12:27.708825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.708714    6481 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 03 14:12:27.784341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.784318    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:27.866692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.866603    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:27.886650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886618    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.886786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886674    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.886786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886713    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.886786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886746    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.886786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886780    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886810    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886845    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886883    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886918    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886946    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.886983    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887048    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887078    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887107    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887140    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887177    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887206    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887259    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887290    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887321    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") pod \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\" (UID: \"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f\") "
Nov 03 14:12:27.887662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887550    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.887918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.887898    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.889109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.889080    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.889654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.889617    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.890681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.890574    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:27.890782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.890705    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:12:27.890952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.890926    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.891070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.891043    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.891154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.891061    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.892262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.892217    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "kube-api-access-95stb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:12:27.892364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.892263    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.892747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.892722    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:12:27.893327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.893195    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.893327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.893291    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.893486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.893359    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.893603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.893570    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.893659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.893628    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.893739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.893721    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.895312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.895286    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:12:27.901541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.901521    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config" pod "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" (UID: "73da9fd2-cd6c-44b8-9bf8-abdfab16d26f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:12:27.982909 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:27.982879    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:12:27.988345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988327    6481 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config-out\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988349    6481 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988358    6481 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-config\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988369    6481 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988380    6481 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-grpc-tls\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988388    6481 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988396    6481 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-tls-assets\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988404    6481 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988415    6481 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-metrics-client-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988424    6481 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988435    6481 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-kube-rbac-proxy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988443    6481 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988453    6481 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-web-config\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988461    6481 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-metrics-client-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988468    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95stb\" (UniqueName: \"kubernetes.io/projected/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-kube-api-access-95stb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988477    6481 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988487    6481 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-db\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988495    6481 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988503    6481 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:27.988612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:27.988511    6481 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:12:28.231767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231739    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf" exitCode=0
Nov 03 14:12:28.231767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231763    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859" exitCode=0
Nov 03 14:12:28.231767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231770    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41" exitCode=0
Nov 03 14:12:28.231767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231779    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757" exitCode=0
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231787    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7" exitCode=0
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231794    6481 generic.go:356] "Generic (PLEG): container finished" podID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116" exitCode=0
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231820    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231819    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"}
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231961    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"}
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.231989    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"}
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.232016    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"}
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.232033    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.232052    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"}
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.232090    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"}
Nov 03 14:12:28.232298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.232127    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73da9fd2-cd6c-44b8-9bf8-abdfab16d26f","Type":"ContainerDied","Data":"7dfd1c0e31c94461414508744092d37835bed5e13d0b9eadb7c30e0f600faab6"}
Nov 03 14:12:28.241103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.241082    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.249744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.249727    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.255908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.255890    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.261973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.261956    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.268488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.268470    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.275100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.275084    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.281356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.281341    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.281599 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.281581    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.281657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.281606    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"} err="failed to get container status \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist"
Nov 03 14:12:28.281657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.281623    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.281863 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.281846    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.281926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.281871    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"} err="failed to get container status \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist"
Nov 03 14:12:28.281926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.281892    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.282123 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.282105    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.282157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282128    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"} err="failed to get container status \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist"
Nov 03 14:12:28.282157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282140    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.282351 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.282333    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.282426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282357    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"} err="failed to get container status \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist"
Nov 03 14:12:28.282426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282376    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.282666 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.282645    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.282703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282673    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"} err="failed to get container status \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist"
Nov 03 14:12:28.282703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282691    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.282893 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.282876    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.282951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282899    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"} err="failed to get container status \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist"
Nov 03 14:12:28.282951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.282917    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.283156 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:28.283141    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.283192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283159    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"} err="failed to get container status \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist"
Nov 03 14:12:28.283192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283171    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.283388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283370    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"} err="failed to get container status \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist"
Nov 03 14:12:28.283449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283388    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.283691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283675    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"} err="failed to get container status \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist"
Nov 03 14:12:28.283691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283690    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.283882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283864    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"} err="failed to get container status \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist"
Nov 03 14:12:28.283951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.283883    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.284079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284065    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"} err="failed to get container status \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist"
Nov 03 14:12:28.284121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284079    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.284283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284262    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"} err="failed to get container status \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist"
Nov 03 14:12:28.284323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284285    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.284478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284458    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"} err="failed to get container status \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist"
Nov 03 14:12:28.284523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284478    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.284668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284649    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"} err="failed to get container status \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist"
Nov 03 14:12:28.284708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284667    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.284859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284841    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"} err="failed to get container status \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist"
Nov 03 14:12:28.284924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.284860    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.285101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285083    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"} err="failed to get container status \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist"
Nov 03 14:12:28.285144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285101    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.285317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285302    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"} err="failed to get container status \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist"
Nov 03 14:12:28.285317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285316    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.285496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285478    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"} err="failed to get container status \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist"
Nov 03 14:12:28.285556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285497    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.285719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285704    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"} err="failed to get container status \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist"
Nov 03 14:12:28.285719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285717    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.285910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285893    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"} err="failed to get container status \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist"
Nov 03 14:12:28.285980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.285911    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.286120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286103    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"} err="failed to get container status \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist"
Nov 03 14:12:28.286166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286121    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.286342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286325    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"} err="failed to get container status \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist"
Nov 03 14:12:28.286406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286343    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.286560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286546    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"} err="failed to get container status \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist"
Nov 03 14:12:28.286603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286560    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.286752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286736    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"} err="failed to get container status \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist"
Nov 03 14:12:28.286818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286752    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.286945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286928    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"} err="failed to get container status \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist"
Nov 03 14:12:28.286989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.286945    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.287132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287115    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"} err="failed to get container status \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist"
Nov 03 14:12:28.287193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287132    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.287349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287333    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"} err="failed to get container status \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist"
Nov 03 14:12:28.287349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287347    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.287581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287564    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"} err="failed to get container status \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist"
Nov 03 14:12:28.287581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287580    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.287793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287777    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"} err="failed to get container status \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist"
Nov 03 14:12:28.287793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.287792    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.288017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288002    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"} err="failed to get container status \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist"
Nov 03 14:12:28.288017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288018    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.288295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288270    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"} err="failed to get container status \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist"
Nov 03 14:12:28.288295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288294    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.288541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288522    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"} err="failed to get container status \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist"
Nov 03 14:12:28.288588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288541    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.288737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288722    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"} err="failed to get container status \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist"
Nov 03 14:12:28.288776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288737    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.288927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288910    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"} err="failed to get container status \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist"
Nov 03 14:12:28.288964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.288926    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.289106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289091    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"} err="failed to get container status \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist"
Nov 03 14:12:28.289106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289105    6481 scope.go:122] "RemoveContainer" containerID="619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"
Nov 03 14:12:28.289321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289295    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf"} err="failed to get container status \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": rpc error: code = NotFound desc = could not find container \"619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf\": container with ID starting with 619373493b6538de801b1feb1e3d9a77eda863c9fa4fe8e8a3ae1409abd16abf not found: ID does not exist"
Nov 03 14:12:28.289370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289321    6481 scope.go:122] "RemoveContainer" containerID="4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"
Nov 03 14:12:28.289507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289492    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859"} err="failed to get container status \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": rpc error: code = NotFound desc = could not find container \"4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859\": container with ID starting with 4b9ead0b68f69bd5bb7de0b88dcf8180174d12173497440ff96b90cf80079859 not found: ID does not exist"
Nov 03 14:12:28.289545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289507    6481 scope.go:122] "RemoveContainer" containerID="d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"
Nov 03 14:12:28.289686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289670    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41"} err="failed to get container status \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": rpc error: code = NotFound desc = could not find container \"d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41\": container with ID starting with d0f9014ac275a3314364af58ed5ccf4651a21eaaa145f1894e3f43bd1cb9ae41 not found: ID does not exist"
Nov 03 14:12:28.289733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289686    6481 scope.go:122] "RemoveContainer" containerID="07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"
Nov 03 14:12:28.289870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289855    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757"} err="failed to get container status \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": rpc error: code = NotFound desc = could not find container \"07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757\": container with ID starting with 07b838de57b5227e60962f7987dd4a11a50ad8ba6ea09358e81336333ad4e757 not found: ID does not exist"
Nov 03 14:12:28.289870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.289869    6481 scope.go:122] "RemoveContainer" containerID="9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"
Nov 03 14:12:28.290055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.290039    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7"} err="failed to get container status \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": rpc error: code = NotFound desc = could not find container \"9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7\": container with ID starting with 9d02b5e72f307cdf9de73265900407e10915ab58dad99222024569fd1fb45cb7 not found: ID does not exist"
Nov 03 14:12:28.290055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.290054    6481 scope.go:122] "RemoveContainer" containerID="64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"
Nov 03 14:12:28.290257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.290241    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116"} err="failed to get container status \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": rpc error: code = NotFound desc = could not find container \"64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116\": container with ID starting with 64f310b715eb3d2730c427eb34964578447e6f8264a031ec1b85ef32e593a116 not found: ID does not exist"
Nov 03 14:12:28.290257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.290255    6481 scope.go:122] "RemoveContainer" containerID="f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"
Nov 03 14:12:28.290513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.290479    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4"} err="failed to get container status \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": rpc error: code = NotFound desc = could not find container \"f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4\": container with ID starting with f33ca1d371a96c01bf952029e6b48ea7a54161be0e5fba781cb1ec3b64812fd4 not found: ID does not exist"
Nov 03 14:12:28.292967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.292950    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:12:28.306311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.306293    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:12:28.334907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.334884    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:12:28.335189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335174    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="thanos-sidecar"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335191    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="thanos-sidecar"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335204    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="init-config-reloader"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335212    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="init-config-reloader"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335240    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335248    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335258    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="prometheus"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335265    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="prometheus"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335283    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-thanos"
Nov 03 14:12:28.335291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335291    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-thanos"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335314    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="config-reloader"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335322    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="config-reloader"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335333    6481 cpu_manager.go:414] "containerMap: removing container" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-web"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335341    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-web"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335406    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-thanos"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335420    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="config-reloader"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335429    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy-web"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335441    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="thanos-sidecar"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335448    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="init-config-reloader"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335457    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="kube-rbac-proxy"
Nov 03 14:12:28.335707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.335466    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" containerName="prometheus"
Nov 03 14:12:28.340153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.340135    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.359347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359321    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 14:12:28.359436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359333    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 14:12:28.359507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359492    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 14:12:28.359733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359720    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 14:12:28.359792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359753    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 14:12:28.359792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359771    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 14:12:28.359792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359755    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 14:12:28.359792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359755    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 14:12:28.359792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359755    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 14:12:28.359946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359805    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9g39dtaijeu5d\""
Nov 03 14:12:28.359978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.359956    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mwv7d\""
Nov 03 14:12:28.363717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.363704    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 14:12:28.364002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.363988    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 14:12:28.364104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.364091    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 14:12:28.378349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.378325    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 14:12:28.378671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.378653    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 14:12:28.379344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.379329    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:12:28.391025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391005    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391035    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391053    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391110    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391142    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391221    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391265    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391356    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-config\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391398    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391430    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391474    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bkwm\" (UniqueName: \"kubernetes.io/projected/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-kube-api-access-8bkwm\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391537    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391575    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391594    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391611    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391635    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391691    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391717    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391762    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.391917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.391796    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492393    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492438    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492455    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492577    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492619    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492654    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-config\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492684    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492711    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492753    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bkwm\" (UniqueName: \"kubernetes.io/projected/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-kube-api-access-8bkwm\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492778    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492802    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492825    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492840    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.492892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492854    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492899    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.492944    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493028    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493052    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493080    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493111    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493141    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493356    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.493680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493657    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.494200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.493900    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.495846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.495382    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.495846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.495415    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.495846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.495577    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.495846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.495797    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-config\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.496575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.496550    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.496716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.496690    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.497474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.497410    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.497474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.497418    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.497826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.497783    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-config-out\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.497826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.497793    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.497826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.497804    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.497958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.497830    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.498793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.498771    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.499139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.499122    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.499525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.499505    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-web-config\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.516010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.515993    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bkwm\" (UniqueName: \"kubernetes.io/projected/4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5-kube-api-access-8bkwm\") pod \"prometheus-k8s-0\" (UID: \"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.664194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.664166    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:28.992658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:28.992624    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 14:12:28.995277 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:12:28.995252    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9d1a17_5793_4dfe_b7d3_b61f8a12ecd5.slice/crio-83c2530a4b124203a16b11af0559f8690d7db1165ed65fee40db956176b1def8 WatchSource:0}: Error finding container 83c2530a4b124203a16b11af0559f8690d7db1165ed65fee40db956176b1def8: Status 404 returned error can't find the container with id 83c2530a4b124203a16b11af0559f8690d7db1165ed65fee40db956176b1def8
Nov 03 14:12:29.238150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:29.238114    6481 generic.go:356] "Generic (PLEG): container finished" podID="4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5" containerID="5be6c5259a1b25e55165668a0d5a1411f7d7bf931e8fc2d95b1dbb6a2aaf36ba" exitCode=0
Nov 03 14:12:29.238594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:29.238196    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerDied","Data":"5be6c5259a1b25e55165668a0d5a1411f7d7bf931e8fc2d95b1dbb6a2aaf36ba"}
Nov 03 14:12:29.238594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:29.238300    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"83c2530a4b124203a16b11af0559f8690d7db1165ed65fee40db956176b1def8"}
Nov 03 14:12:30.064997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.064964    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73da9fd2-cd6c-44b8-9bf8-abdfab16d26f" path="/var/lib/kubelet/pods/73da9fd2-cd6c-44b8-9bf8-abdfab16d26f/volumes"
Nov 03 14:12:30.245614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.245582    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"aec71c971e142b266b024ac2fedbc365f85a3e7191aa6bc2c4b73d8e187c0b4c"}
Nov 03 14:12:30.245984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.245645    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"f6bb59f3ca035e8f2b0ce09c712e29e0f92236fa6d5d050b456f0f3c2e6df5b9"}
Nov 03 14:12:30.245984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.245673    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"a1c871d417072d77cebad2fbbd27510f364541f70c3a1664db7eaea8987921cb"}
Nov 03 14:12:30.245984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.245698    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"d3df482e97b9ceed76d38aa7f263e948da18136d55ecac240d6db31202520547"}
Nov 03 14:12:30.245984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.245730    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"f897de0b87b8932f5c2a19b690c965292ca522ade19fbffa611e13a50dd0495a"}
Nov 03 14:12:30.245984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.245752    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5","Type":"ContainerStarted","Data":"7861bd3349af91f997954d862eb22d373c88a12cb309d929bfdf384bdf0d40b2"}
Nov 03 14:12:30.315288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:30.315186    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.2989478070000002 podStartE2EDuration="2.315172995s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.225188ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:30.314718067 +0000 UTC m=+28.364749280" watchObservedRunningTime="2026-11-03 14:12:30.315172995 +0000 UTC m=+28.365204189"
Nov 03 14:12:33.664895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:33.664853    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:12:34.851486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:34.851457    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s7z9p"
Nov 03 14:12:42.609043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:42.609002    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:42.827497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:42.827463    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:55.520970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:12:55.520934    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:12:57.983596 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:12:57.983556    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:13:06.563839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:13:06.563806    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:13:27.461185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:13:27.461149    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:13:27.476201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:13:27.476178    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:13:27.984164 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:13:27.984128    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:13:28.458394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:13:28.458363    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 14:13:55.633000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:13:55.632916    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:13:57.984721 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:13:57.984687    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:14:27.077636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:14:27.077601    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:14:27.983596 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:14:27.983559    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:14:55.751140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:14:55.751105    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:14:57.984527 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:14:57.984485    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:15:06.559315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:06.559277    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:10.440733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:10.440698    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:10.656388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:10.656354    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:10.869500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:10.869428    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:12.166279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:12.166243    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:12.385784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:12.385751    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:13.655148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:13.655069    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:15.430439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.430404    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"]
Nov 03 14:15:15.433827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.433803    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.463324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.463300    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 14:15:15.463460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.463357    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 14:15:15.463460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.463414    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-m5hlg\""
Nov 03 14:15:15.464419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.464403    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"]
Nov 03 14:15:15.551665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.551634    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.551821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.551672    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.551821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.551780    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9gw\" (UniqueName: \"kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.652829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.652777    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9gw\" (UniqueName: \"kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.652966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.652841    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.652966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.652866    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.653213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.653194    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.653267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.653217    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.675202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.675180    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9gw\" (UniqueName: \"kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:15.748911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:15.748841    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:16.044431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:16.044402    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"]
Nov 03 14:15:16.047059 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:15:16.047036    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0df4b14_838b_4ecb_96dc_e6f92d27edb5.slice/crio-db5e90958bb4b57fe66dd2c808d439f8d14c43c0c8cdac9f52f5ca8a7c158e2a WatchSource:0}: Error finding container db5e90958bb4b57fe66dd2c808d439f8d14c43c0c8cdac9f52f5ca8a7c158e2a: Status 404 returned error can't find the container with id db5e90958bb4b57fe66dd2c808d439f8d14c43c0c8cdac9f52f5ca8a7c158e2a
Nov 03 14:15:16.049516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:16.049498    6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:15:16.669526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:16.669482    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:16.793507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:16.793472    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg" event={"ID":"a0df4b14-838b-4ecb-96dc-e6f92d27edb5","Type":"ContainerStarted","Data":"db5e90958bb4b57fe66dd2c808d439f8d14c43c0c8cdac9f52f5ca8a7c158e2a"}
Nov 03 14:15:16.897978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:16.897940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:17.447134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.447098    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"]
Nov 03 14:15:17.450572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.450537    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.474695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.474664    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\""
Nov 03 14:15:17.476302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.476161    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 14:15:17.481643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.481603    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 14:15:17.481934 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.481914    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 14:15:17.483180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.483149    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"]
Nov 03 14:15:17.567739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.567705    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"]
Nov 03 14:15:17.569740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.569704    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.569869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.569825    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.569938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.569914    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.570001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.569951    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggtb5\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.570057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.570000    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.570100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.570077    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.570211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.570197    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.570336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.570246    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.570435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.570418    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.571460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.571440    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.588620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.588596    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-tqwvg\""
Nov 03 14:15:17.591641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.591616    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"]
Nov 03 14:15:17.628578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.628549    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"]
Nov 03 14:15:17.632363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.632338    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.648918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.648894    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:17.653524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.653505    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-qbf6f\""
Nov 03 14:15:17.662580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.662554    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"]
Nov 03 14:15:17.671250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671201    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671266    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggtb5\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671309    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671342    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671367    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671394    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671428    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671482    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ln5f\" (UniqueName: \"kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671506    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671534    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671561    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671597    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.671624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671623    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671645    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671668    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671692    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671710    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671736    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671757    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671775    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.672268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.671790    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb94g\" (UniqueName: \"kubernetes.io/projected/0862f426-a7ac-4f13-a39f-0e0925aa4c07-kube-api-access-tb94g\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.672676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.672375    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.673104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.673079    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.673412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.673380    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.674097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.674070    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.674192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.674174    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.676641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.676423    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.676641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.676599    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.697818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.697759    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.712634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.712606    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggtb5\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-675z5\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.766419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.766391    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:17.772532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772511    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.772624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772543    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.772624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772563    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.772920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772861    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772931    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772954    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb94g\" (UniqueName: \"kubernetes.io/projected/0862f426-a7ac-4f13-a39f-0e0925aa4c07-kube-api-access-tb94g\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.772977    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.773012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773009    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773034    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773068    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.773281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773113    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ln5f\" (UniqueName: \"kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.773281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773140    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773166    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773196    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773356    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773412    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.773589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773414    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.773934 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.773888    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.774085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.774068    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.775296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.775274    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.776030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.776008    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.800588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.800530    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ln5f\" (UniqueName: \"kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.801289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.800987    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0862f426-a7ac-4f13-a39f-0e0925aa4c07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.801289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.801256    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb94g\" (UniqueName: \"kubernetes.io/projected/0862f426-a7ac-4f13-a39f-0e0925aa4c07-kube-api-access-tb94g\") pod \"openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t\" (UID: \"0862f426-a7ac-4f13-a39f-0e0925aa4c07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:17.869855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.869767    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:17.889873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.889843    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:17.947317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:17.947242    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:18.085527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:18.085495    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"]
Nov 03 14:15:18.089494 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:15:18.089464    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053a3628_1b59_4617_801f_21d29607180b.slice/crio-98539b3980adc671866587a09040494b3dbb738d4c0dd463058c023550b8baa5 WatchSource:0}: Error finding container 98539b3980adc671866587a09040494b3dbb738d4c0dd463058c023550b8baa5: Status 404 returned error can't find the container with id 98539b3980adc671866587a09040494b3dbb738d4c0dd463058c023550b8baa5
Nov 03 14:15:18.189679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:18.189539    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"]
Nov 03 14:15:18.191764 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:15:18.191738    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e2b9de_1b9a_4a85_a817_182c140b922e.slice/crio-219f5d4c4ee9398b4f55b466e9703fecafc24a9edd2572deb3e20d6659027bf3 WatchSource:0}: Error finding container 219f5d4c4ee9398b4f55b466e9703fecafc24a9edd2572deb3e20d6659027bf3: Status 404 returned error can't find the container with id 219f5d4c4ee9398b4f55b466e9703fecafc24a9edd2572deb3e20d6659027bf3
Nov 03 14:15:18.275477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:18.275388    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"]
Nov 03 14:15:18.278687 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:15:18.278658    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0862f426_a7ac_4f13_a39f_0e0925aa4c07.slice/crio-5f38d6833e2ea70d401269adedc9e0688631bac7cbf1fccad3774ea00ae9c098 WatchSource:0}: Error finding container 5f38d6833e2ea70d401269adedc9e0688631bac7cbf1fccad3774ea00ae9c098: Status 404 returned error can't find the container with id 5f38d6833e2ea70d401269adedc9e0688631bac7cbf1fccad3774ea00ae9c098
Nov 03 14:15:18.804665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:18.804619    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f" event={"ID":"d5e2b9de-1b9a-4a85-a817-182c140b922e","Type":"ContainerStarted","Data":"219f5d4c4ee9398b4f55b466e9703fecafc24a9edd2572deb3e20d6659027bf3"}
Nov 03 14:15:18.806417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:18.806381    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" event={"ID":"053a3628-1b59-4617-801f-21d29607180b","Type":"ContainerStarted","Data":"98539b3980adc671866587a09040494b3dbb738d4c0dd463058c023550b8baa5"}
Nov 03 14:15:18.807913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:18.807879    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t" event={"ID":"0862f426-a7ac-4f13-a39f-0e0925aa4c07","Type":"ContainerStarted","Data":"5f38d6833e2ea70d401269adedc9e0688631bac7cbf1fccad3774ea00ae9c098"}
Nov 03 14:15:20.942867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:20.942833    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:21.568815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:21.568785    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:15:21.568918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:21.568853    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:15:21.568918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:21.568890    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:15:21.824362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:21.824269    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" event={"ID":"053a3628-1b59-4617-801f-21d29607180b","Type":"ContainerStarted","Data":"becef13b733565201b43de0616646848747ca94c2390ebb7f29f56b270d093d4"}
Nov 03 14:15:21.884247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:21.884179    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podStartSLOduration=1.408098083 podStartE2EDuration="4.884162367s" totalImagesPullingTime="3.476064284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:21.883302014 +0000 UTC m=+199.933333200" watchObservedRunningTime="2026-11-03 14:15:21.884162367 +0000 UTC m=+199.934193550"
Nov 03 14:15:22.766665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:22.766636    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:22.772110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:22.772086    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:22.829832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:22.829787    6481 generic.go:356] "Generic (PLEG): container finished" podID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerID="db7d90a0268c2097c0363e1e6a351068595dae5bf35344fd38348d8e1e236dff" exitCode=0
Nov 03 14:15:22.830186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:22.829916    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg" event={"ID":"a0df4b14-838b-4ecb-96dc-e6f92d27edb5","Type":"ContainerDied","Data":"db7d90a0268c2097c0363e1e6a351068595dae5bf35344fd38348d8e1e236dff"}
Nov 03 14:15:22.830794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:22.830325    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:22.831543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:22.831521    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:15:24.839409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:24.839327    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerID="da7adc772e75c4b7b07631cdbfaf13fb6d170ae1d4575e27d460508607abc890" exitCode=0
Nov 03 14:15:24.839855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:24.839420    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f" event={"ID":"d5e2b9de-1b9a-4a85-a817-182c140b922e","Type":"ContainerDied","Data":"da7adc772e75c4b7b07631cdbfaf13fb6d170ae1d4575e27d460508607abc890"}
Nov 03 14:15:25.845114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:25.845079    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerID="680e984d74670d325e90b8f0657b92f06997997e89e64ff153e9bfc94413566a" exitCode=0
Nov 03 14:15:25.845584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:25.845163    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f" event={"ID":"d5e2b9de-1b9a-4a85-a817-182c140b922e","Type":"ContainerDied","Data":"680e984d74670d325e90b8f0657b92f06997997e89e64ff153e9bfc94413566a"}
Nov 03 14:15:25.847077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:25.847052    6481 generic.go:356] "Generic (PLEG): container finished" podID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerID="7f5d3a4284b8c19dfcc2e6e245ca3e79d3d500ecc64db4d987832ebc49550466" exitCode=0
Nov 03 14:15:25.847177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:25.847082    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg" event={"ID":"a0df4b14-838b-4ecb-96dc-e6f92d27edb5","Type":"ContainerDied","Data":"7f5d3a4284b8c19dfcc2e6e245ca3e79d3d500ecc64db4d987832ebc49550466"}
Nov 03 14:15:27.994733 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:15:27.994700    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:15:32.879675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:32.879637    6481 generic.go:356] "Generic (PLEG): container finished" podID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerID="b00aeb8ba16fca241396f625a4fe81df4ee4ab6a32241ff52bd8d544a3d536ed" exitCode=0
Nov 03 14:15:32.880128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:32.879724    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg" event={"ID":"a0df4b14-838b-4ecb-96dc-e6f92d27edb5","Type":"ContainerDied","Data":"b00aeb8ba16fca241396f625a4fe81df4ee4ab6a32241ff52bd8d544a3d536ed"}
Nov 03 14:15:32.881673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:32.881651    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerID="79d9207de03e54517f5fcaf838d2d04576e8ecd3a13ca35211c1ed9215a20efe" exitCode=0
Nov 03 14:15:32.881792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:32.881683    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f" event={"ID":"d5e2b9de-1b9a-4a85-a817-182c140b922e","Type":"ContainerDied","Data":"79d9207de03e54517f5fcaf838d2d04576e8ecd3a13ca35211c1ed9215a20efe"}
Nov 03 14:15:34.022980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.022893    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:34.036284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.036265    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:34.120794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.120763    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle\") pod \"d5e2b9de-1b9a-4a85-a817-182c140b922e\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") "
Nov 03 14:15:34.120963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.120815    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw\" (UniqueName: \"kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw\") pod \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") "
Nov 03 14:15:34.120963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.120875    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util\") pod \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") "
Nov 03 14:15:34.120963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.120903    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f\" (UniqueName: \"kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f\") pod \"d5e2b9de-1b9a-4a85-a817-182c140b922e\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") "
Nov 03 14:15:34.121131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.121021    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util\") pod \"d5e2b9de-1b9a-4a85-a817-182c140b922e\" (UID: \"d5e2b9de-1b9a-4a85-a817-182c140b922e\") "
Nov 03 14:15:34.121131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.121107    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle\") pod \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\" (UID: \"a0df4b14-838b-4ecb-96dc-e6f92d27edb5\") "
Nov 03 14:15:34.121779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.121738    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle" pod "a0df4b14-838b-4ecb-96dc-e6f92d27edb5" (UID: "a0df4b14-838b-4ecb-96dc-e6f92d27edb5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:15:34.121937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.121912    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle" pod "d5e2b9de-1b9a-4a85-a817-182c140b922e" (UID: "d5e2b9de-1b9a-4a85-a817-182c140b922e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:15:34.123727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.123699    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f" pod "d5e2b9de-1b9a-4a85-a817-182c140b922e" (UID: "d5e2b9de-1b9a-4a85-a817-182c140b922e"). InnerVolumeSpecName "kube-api-access-5ln5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:15:34.123727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.123709    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw" pod "a0df4b14-838b-4ecb-96dc-e6f92d27edb5" (UID: "a0df4b14-838b-4ecb-96dc-e6f92d27edb5"). InnerVolumeSpecName "kube-api-access-4l9gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:15:34.125564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.125533    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util" pod "d5e2b9de-1b9a-4a85-a817-182c140b922e" (UID: "d5e2b9de-1b9a-4a85-a817-182c140b922e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:15:34.127584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.127558    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util" pod "a0df4b14-838b-4ecb-96dc-e6f92d27edb5" (UID: "a0df4b14-838b-4ecb-96dc-e6f92d27edb5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:15:34.222138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.222109    6481 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-util\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:15:34.222138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.222137    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ln5f\" (UniqueName: \"kubernetes.io/projected/d5e2b9de-1b9a-4a85-a817-182c140b922e-kube-api-access-5ln5f\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:15:34.222346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.222149    6481 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-util\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:15:34.222346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.222161    6481 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-bundle\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:15:34.222346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.222173    6481 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5e2b9de-1b9a-4a85-a817-182c140b922e-bundle\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:15:34.222346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.222183    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9gw\" (UniqueName: \"kubernetes.io/projected/a0df4b14-838b-4ecb-96dc-e6f92d27edb5-kube-api-access-4l9gw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:15:34.889526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.889491    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f"
Nov 03 14:15:34.889526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.889502    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ap276f" event={"ID":"d5e2b9de-1b9a-4a85-a817-182c140b922e","Type":"ContainerDied","Data":"219f5d4c4ee9398b4f55b466e9703fecafc24a9edd2572deb3e20d6659027bf3"}
Nov 03 14:15:34.889790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.889554    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="219f5d4c4ee9398b4f55b466e9703fecafc24a9edd2572deb3e20d6659027bf3"
Nov 03 14:15:34.891302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.891279    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg" event={"ID":"a0df4b14-838b-4ecb-96dc-e6f92d27edb5","Type":"ContainerDied","Data":"db5e90958bb4b57fe66dd2c808d439f8d14c43c0c8cdac9f52f5ca8a7c158e2a"}
Nov 03 14:15:34.891414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.891311    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db5e90958bb4b57fe66dd2c808d439f8d14c43c0c8cdac9f52f5ca8a7c158e2a"
Nov 03 14:15:34.891414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:34.891316    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644kdvmg"
Nov 03 14:15:36.840155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:36.840119    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:38.143823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.143789    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:38.166144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166113    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"]
Nov 03 14:15:38.166476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166461    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="pull"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166477    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="pull"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166488    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="extract"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166494    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="extract"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166501    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="util"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166505    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="util"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166522    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="pull"
Nov 03 14:15:38.166530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166526    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="pull"
Nov 03 14:15:38.166739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166540    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="util"
Nov 03 14:15:38.166739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166545    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="util"
Nov 03 14:15:38.166739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166550    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="extract"
Nov 03 14:15:38.166739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166555    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="extract"
Nov 03 14:15:38.166739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166607    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e2b9de-1b9a-4a85-a817-182c140b922e" containerName="extract"
Nov 03 14:15:38.166739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.166617    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0df4b14-838b-4ecb-96dc-e6f92d27edb5" containerName="extract"
Nov 03 14:15:38.170781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.170765    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.206569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.206548    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 14:15:38.206569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.206551    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 14:15:38.206769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.206601    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-f588q\""
Nov 03 14:15:38.207361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.207340    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"]
Nov 03 14:15:38.255688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.255664    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccp47\" (UniqueName: \"kubernetes.io/projected/20716d70-8a0b-4300-a5af-60cdce599bbd-kube-api-access-ccp47\") pod \"jobset-operator-674cd59dc5-mk4zz\" (UID: \"20716d70-8a0b-4300-a5af-60cdce599bbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.255771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.255698    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20716d70-8a0b-4300-a5af-60cdce599bbd-tmp\") pod \"jobset-operator-674cd59dc5-mk4zz\" (UID: \"20716d70-8a0b-4300-a5af-60cdce599bbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.356145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.356115    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccp47\" (UniqueName: \"kubernetes.io/projected/20716d70-8a0b-4300-a5af-60cdce599bbd-kube-api-access-ccp47\") pod \"jobset-operator-674cd59dc5-mk4zz\" (UID: \"20716d70-8a0b-4300-a5af-60cdce599bbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.356307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.356158    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20716d70-8a0b-4300-a5af-60cdce599bbd-tmp\") pod \"jobset-operator-674cd59dc5-mk4zz\" (UID: \"20716d70-8a0b-4300-a5af-60cdce599bbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.356626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.356608    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/20716d70-8a0b-4300-a5af-60cdce599bbd-tmp\") pod \"jobset-operator-674cd59dc5-mk4zz\" (UID: \"20716d70-8a0b-4300-a5af-60cdce599bbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.378096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.378076    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccp47\" (UniqueName: \"kubernetes.io/projected/20716d70-8a0b-4300-a5af-60cdce599bbd-kube-api-access-ccp47\") pod \"jobset-operator-674cd59dc5-mk4zz\" (UID: \"20716d70-8a0b-4300-a5af-60cdce599bbd\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.490903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.490873    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"
Nov 03 14:15:38.772108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.772077    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:15:38.772169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.772145    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:15:38.772207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.772175    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:15:38.783491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.783472    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz"]
Nov 03 14:15:38.785850 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:15:38.785828    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20716d70_8a0b_4300_a5af_60cdce599bbd.slice/crio-3215f5328118025731c604e31c7270740a61eac1a788723611645fb96cef68e8 WatchSource:0}: Error finding container 3215f5328118025731c604e31c7270740a61eac1a788723611645fb96cef68e8: Status 404 returned error can't find the container with id 3215f5328118025731c604e31c7270740a61eac1a788723611645fb96cef68e8
Nov 03 14:15:38.908492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.908456    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t" event={"ID":"0862f426-a7ac-4f13-a39f-0e0925aa4c07","Type":"ContainerStarted","Data":"be6a953a72d901c85978c984ed4245e0eb875b8635b32961084cfaa315027e32"}
Nov 03 14:15:38.909932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.909906    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz" event={"ID":"20716d70-8a0b-4300-a5af-60cdce599bbd","Type":"ContainerStarted","Data":"3215f5328118025731c604e31c7270740a61eac1a788723611645fb96cef68e8"}
Nov 03 14:15:38.948350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.948317    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:38.954129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:38.954083    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t" podStartSLOduration=1.4639791500000001 podStartE2EDuration="21.954069688s" totalImagesPullingTime="20.490090538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:38.953049904 +0000 UTC m=+217.003081099" watchObservedRunningTime="2026-11-03 14:15:38.954069688 +0000 UTC m=+217.004100870"
Nov 03 14:15:39.953027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:39.953000    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:40.919058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:40.919023    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:40.920035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:40.920011    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t"
Nov 03 14:15:41.923729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:41.923697    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz" event={"ID":"20716d70-8a0b-4300-a5af-60cdce599bbd","Type":"ContainerStarted","Data":"b617fc28dbf70b4ecf218692076c5d209d1ece481bcffc18c96bd923fb08e153"}
Nov 03 14:15:41.967613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:41.967571    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mk4zz" podStartSLOduration=1.666264676 podStartE2EDuration="3.967556977s" totalImagesPullingTime="2.301292301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:41.96694653 +0000 UTC m=+220.016977714" watchObservedRunningTime="2026-11-03 14:15:41.967556977 +0000 UTC m=+220.017588160"
Nov 03 14:15:50.398795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:50.398761    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:52.205104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:52.205073    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:52.409845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:52.409813    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:54.811920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:54.811889    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:55.616961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:55.616929    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:55.865439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:55.865409    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:56.427321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:56.427292    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:15:57.985044 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:15:57.985007    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:15:58.634763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:15:58.634732    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:00.928984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:00.928952    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:01.134033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:01.133998    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:01.662401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:01.662371    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:01.886354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:01.886323    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:01.957841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:01.957756    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"]
Nov 03 14:16:01.967751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:01.967727    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:01.988761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:01.988736    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"]
Nov 03 14:16:02.000776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.000726    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 14:16:02.000776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.000746    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 14:16:02.000776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.000786    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k2kgk\""
Nov 03 14:16:02.049707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.049678    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102b10fa-0c38-40bb-9c27-22cad490acaf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9z5kt\" (UID: \"102b10fa-0c38-40bb-9c27-22cad490acaf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.049859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.049722    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvgb\" (UniqueName: \"kubernetes.io/projected/102b10fa-0c38-40bb-9c27-22cad490acaf-kube-api-access-ggvgb\") pod \"cert-manager-webhook-5dbbbf75cc-9z5kt\" (UID: \"102b10fa-0c38-40bb-9c27-22cad490acaf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.109056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.109015    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:02.150551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.150524    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102b10fa-0c38-40bb-9c27-22cad490acaf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9z5kt\" (UID: \"102b10fa-0c38-40bb-9c27-22cad490acaf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.150725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.150571    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvgb\" (UniqueName: \"kubernetes.io/projected/102b10fa-0c38-40bb-9c27-22cad490acaf-kube-api-access-ggvgb\") pod \"cert-manager-webhook-5dbbbf75cc-9z5kt\" (UID: \"102b10fa-0c38-40bb-9c27-22cad490acaf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.176170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.176136    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/102b10fa-0c38-40bb-9c27-22cad490acaf-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-9z5kt\" (UID: \"102b10fa-0c38-40bb-9c27-22cad490acaf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.188511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.188493    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 14:16:02.208853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.208811    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 14:16:02.216907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.216881    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvgb\" (UniqueName: \"kubernetes.io/projected/102b10fa-0c38-40bb-9c27-22cad490acaf-kube-api-access-ggvgb\") pod \"cert-manager-webhook-5dbbbf75cc-9z5kt\" (UID: \"102b10fa-0c38-40bb-9c27-22cad490acaf\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.294968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.294948    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-k2kgk\""
Nov 03 14:16:02.301101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.301085    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:02.606347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.606274    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:02.630752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.630733    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"]
Nov 03 14:16:02.633077 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:02.633049    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod102b10fa_0c38_40bb_9c27_22cad490acaf.slice/crio-7f1268549e65dcc448982bf228368955ee354b00b3b8cac872df5b58dbb5a1c0 WatchSource:0}: Error finding container 7f1268549e65dcc448982bf228368955ee354b00b3b8cac872df5b58dbb5a1c0: Status 404 returned error can't find the container with id 7f1268549e65dcc448982bf228368955ee354b00b3b8cac872df5b58dbb5a1c0
Nov 03 14:16:02.846307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:02.846268    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:03.005272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.005218    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt" event={"ID":"102b10fa-0c38-40bb-9c27-22cad490acaf","Type":"ContainerStarted","Data":"7f1268549e65dcc448982bf228368955ee354b00b3b8cac872df5b58dbb5a1c0"}
Nov 03 14:16:03.414985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.414952    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:03.549495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.549464    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"]
Nov 03 14:16:03.581742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.581716    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"]
Nov 03 14:16:03.581909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.581828    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.599625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.599606    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-r8dbh\""
Nov 03 14:16:03.662024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.661996    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea196d1d-fa55-4157-8245-458669fec7cc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wzr8g\" (UID: \"ea196d1d-fa55-4157-8245-458669fec7cc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.662140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.662039    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p577\" (UniqueName: \"kubernetes.io/projected/ea196d1d-fa55-4157-8245-458669fec7cc-kube-api-access-5p577\") pod \"cert-manager-cainjector-5c57bbd4fc-wzr8g\" (UID: \"ea196d1d-fa55-4157-8245-458669fec7cc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.763558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.763490    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea196d1d-fa55-4157-8245-458669fec7cc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wzr8g\" (UID: \"ea196d1d-fa55-4157-8245-458669fec7cc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.763558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.763526    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p577\" (UniqueName: \"kubernetes.io/projected/ea196d1d-fa55-4157-8245-458669fec7cc-kube-api-access-5p577\") pod \"cert-manager-cainjector-5c57bbd4fc-wzr8g\" (UID: \"ea196d1d-fa55-4157-8245-458669fec7cc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.785898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.785868    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ea196d1d-fa55-4157-8245-458669fec7cc-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-wzr8g\" (UID: \"ea196d1d-fa55-4157-8245-458669fec7cc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.786014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.785996    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p577\" (UniqueName: \"kubernetes.io/projected/ea196d1d-fa55-4157-8245-458669fec7cc-kube-api-access-5p577\") pod \"cert-manager-cainjector-5c57bbd4fc-wzr8g\" (UID: \"ea196d1d-fa55-4157-8245-458669fec7cc\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:03.905806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:03.905774    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"
Nov 03 14:16:04.213677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:04.213647    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g"]
Nov 03 14:16:04.216633 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:04.216603    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea196d1d_fa55_4157_8245_458669fec7cc.slice/crio-497eca5235dde0f66175732081d19340f6f59bd7c39e831a295df4814c5a7d3f WatchSource:0}: Error finding container 497eca5235dde0f66175732081d19340f6f59bd7c39e831a295df4814c5a7d3f: Status 404 returned error can't find the container with id 497eca5235dde0f66175732081d19340f6f59bd7c39e831a295df4814c5a7d3f
Nov 03 14:16:05.014921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:05.014884    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g" event={"ID":"ea196d1d-fa55-4157-8245-458669fec7cc","Type":"ContainerStarted","Data":"497eca5235dde0f66175732081d19340f6f59bd7c39e831a295df4814c5a7d3f"}
Nov 03 14:16:06.061941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:06.061907    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:06.272140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:06.272069    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:07.855639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:07.855545    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:08.029687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.029650    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt" event={"ID":"102b10fa-0c38-40bb-9c27-22cad490acaf","Type":"ContainerStarted","Data":"71d8abd75fa0bbf3bb265a1fee7f7cf7b5466a75ebccbcaa5490015775c52ece"}
Nov 03 14:16:08.029871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.029805    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:08.031395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.031372    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g" event={"ID":"ea196d1d-fa55-4157-8245-458669fec7cc","Type":"ContainerStarted","Data":"f0352614101280f763c302c4e6a8f91bd04b37ac2806a8c89901d4af50c18e88"}
Nov 03 14:16:08.073807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.073754    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt" podStartSLOduration=2.1547182129999998 podStartE2EDuration="7.073740812s" totalImagesPullingTime="4.919022599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:08.072622369 +0000 UTC m=+246.122653551" watchObservedRunningTime="2026-11-03 14:16:08.073740812 +0000 UTC m=+246.123771994"
Nov 03 14:16:08.090886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.090859    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:08.119296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.119199    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-wzr8g" podStartSLOduration=1.927068043 podStartE2EDuration="5.119187307s" totalImagesPullingTime="3.192119264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:08.118877 +0000 UTC m=+246.168908182" watchObservedRunningTime="2026-11-03 14:16:08.119187307 +0000 UTC m=+246.169218490"
Nov 03 14:16:08.319982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.319951    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:08.578618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.578591    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:08.790447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:08.790419    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:09.461008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.460973    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:09.594139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.594107    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"]
Nov 03 14:16:09.596697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.596676    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.631479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.631448    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"]
Nov 03 14:16:09.672411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.672388    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 14:16:09.672411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.672395    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 14:16:09.676554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.676532    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:09.713139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.713089    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 14:16:09.715978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.715961    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 14:16:09.716044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.716037    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-m8l2v\""
Nov 03 14:16:09.716098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.716086    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 14:16:09.717283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.717265    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.717328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.717295    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.717328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.717316    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5e427ca-47f5-4697-9dd1-3a8b840fc499-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.717451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.717437    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.717503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.717491    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e427ca-47f5-4697-9dd1-3a8b840fc499-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.717557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.717543    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9qm\" (UniqueName: \"kubernetes.io/projected/b5e427ca-47f5-4697-9dd1-3a8b840fc499-kube-api-access-6f9qm\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.818356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.818315    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.818356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.818359    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5e427ca-47f5-4697-9dd1-3a8b840fc499-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.818526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.818406    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.818526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.818471    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e427ca-47f5-4697-9dd1-3a8b840fc499-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.818526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.818496    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9qm\" (UniqueName: \"kubernetes.io/projected/b5e427ca-47f5-4697-9dd1-3a8b840fc499-kube-api-access-6f9qm\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.818733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.818568    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.819032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.819005    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5e427ca-47f5-4697-9dd1-3a8b840fc499-manifests\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.821262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.821219    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5e427ca-47f5-4697-9dd1-3a8b840fc499-tmp\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.821349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.821317    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-webhook-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.821866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.821846    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-apiservice-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.821974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.821954    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e427ca-47f5-4697-9dd1-3a8b840fc499-metrics-cert\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.841251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.841201    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9qm\" (UniqueName: \"kubernetes.io/projected/b5e427ca-47f5-4697-9dd1-3a8b840fc499-kube-api-access-6f9qm\") pod \"opendatahub-operator-controller-manager-867995fc99-zzfz8\" (UID: \"b5e427ca-47f5-4697-9dd1-3a8b840fc499\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:09.906968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.906938    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:09.909888 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:09.909869    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:10.136499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:10.136418    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:10.266488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:10.266457    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"]
Nov 03 14:16:10.279805 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:10.279779    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e427ca_47f5_4697_9dd1_3a8b840fc499.slice/crio-3c5bf92e56a7e8042f035e4d88bad8745a893b9a74080440bc4bd2996a3af909 WatchSource:0}: Error finding container 3c5bf92e56a7e8042f035e4d88bad8745a893b9a74080440bc4bd2996a3af909: Status 404 returned error can't find the container with id 3c5bf92e56a7e8042f035e4d88bad8745a893b9a74080440bc4bd2996a3af909
Nov 03 14:16:10.363126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:10.363099    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:10.592131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:10.592104    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:11.045754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:11.045717    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8" event={"ID":"b5e427ca-47f5-4697-9dd1-3a8b840fc499","Type":"ContainerStarted","Data":"3c5bf92e56a7e8042f035e4d88bad8745a893b9a74080440bc4bd2996a3af909"}
Nov 03 14:16:11.614566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:11.614536    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:13.629457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:13.629427    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:13.832719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:13.832647    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:13.927665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:13.927629    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-d2m4k"]
Nov 03 14:16:13.930848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:13.930828    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:13.959332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:13.959311    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lfl6t\""
Nov 03 14:16:13.960568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:13.960551    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-d2m4k"]
Nov 03 14:16:14.037860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.037834    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-9z5kt"
Nov 03 14:16:14.059806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.059775    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0102d7a8-7048-43f4-86cc-06c3573900db-bound-sa-token\") pod \"cert-manager-597b95cb84-d2m4k\" (UID: \"0102d7a8-7048-43f4-86cc-06c3573900db\") " pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.059965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.059913    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcqc9\" (UniqueName: \"kubernetes.io/projected/0102d7a8-7048-43f4-86cc-06c3573900db-kube-api-access-vcqc9\") pod \"cert-manager-597b95cb84-d2m4k\" (UID: \"0102d7a8-7048-43f4-86cc-06c3573900db\") " pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.060326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.060300    6481 generic.go:356] "Generic (PLEG): container finished" podID="b5e427ca-47f5-4697-9dd1-3a8b840fc499" containerID="ab409b6aa0072d128d3a0c25bc2cde33704f9156548f272e73f3bf2041778824" exitCode=0
Nov 03 14:16:14.064451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.064425    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8" event={"ID":"b5e427ca-47f5-4697-9dd1-3a8b840fc499","Type":"ContainerDied","Data":"ab409b6aa0072d128d3a0c25bc2cde33704f9156548f272e73f3bf2041778824"}
Nov 03 14:16:14.161148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.161122    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0102d7a8-7048-43f4-86cc-06c3573900db-bound-sa-token\") pod \"cert-manager-597b95cb84-d2m4k\" (UID: \"0102d7a8-7048-43f4-86cc-06c3573900db\") " pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.161402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.161378    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcqc9\" (UniqueName: \"kubernetes.io/projected/0102d7a8-7048-43f4-86cc-06c3573900db-kube-api-access-vcqc9\") pod \"cert-manager-597b95cb84-d2m4k\" (UID: \"0102d7a8-7048-43f4-86cc-06c3573900db\") " pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.187865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.187839    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0102d7a8-7048-43f4-86cc-06c3573900db-bound-sa-token\") pod \"cert-manager-597b95cb84-d2m4k\" (UID: \"0102d7a8-7048-43f4-86cc-06c3573900db\") " pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.192411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.192392    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcqc9\" (UniqueName: \"kubernetes.io/projected/0102d7a8-7048-43f4-86cc-06c3573900db-kube-api-access-vcqc9\") pod \"cert-manager-597b95cb84-d2m4k\" (UID: \"0102d7a8-7048-43f4-86cc-06c3573900db\") " pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.241990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.241968    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-d2m4k"
Nov 03 14:16:14.536198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:14.536172    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-d2m4k"]
Nov 03 14:16:14.538377 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:14.538352    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0102d7a8_7048_43f4_86cc_06c3573900db.slice/crio-5707c578d83dd4c2ea9ba957feab27c249168ad1b9ca3f6d49f18d060f1976ad WatchSource:0}: Error finding container 5707c578d83dd4c2ea9ba957feab27c249168ad1b9ca3f6d49f18d060f1976ad: Status 404 returned error can't find the container with id 5707c578d83dd4c2ea9ba957feab27c249168ad1b9ca3f6d49f18d060f1976ad
Nov 03 14:16:15.066345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:15.066312    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8" event={"ID":"b5e427ca-47f5-4697-9dd1-3a8b840fc499","Type":"ContainerStarted","Data":"ec2adb3961bdac1a19fb0f8b7caeda2d56dd67a9a894b96a3ebf5f79560fd928"}
Nov 03 14:16:15.066799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:15.066384    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:15.067894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:15.067871    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-d2m4k" event={"ID":"0102d7a8-7048-43f4-86cc-06c3573900db","Type":"ContainerStarted","Data":"4f8ad029450840f0657884f4db66102b121080f9a9488c95dbea7de5808c85df"}
Nov 03 14:16:15.067999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:15.067901    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-d2m4k" event={"ID":"0102d7a8-7048-43f4-86cc-06c3573900db","Type":"ContainerStarted","Data":"5707c578d83dd4c2ea9ba957feab27c249168ad1b9ca3f6d49f18d060f1976ad"}
Nov 03 14:16:15.133465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:15.133422    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8" podStartSLOduration=2.6822487280000002 podStartE2EDuration="6.133410018s" totalImagesPullingTime="3.007871383s" totalInitContainerRuntime="443.289907ms" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:15.131421391 +0000 UTC m=+253.181452575" watchObservedRunningTime="2026-11-03 14:16:15.133410018 +0000 UTC m=+253.183441201"
Nov 03 14:16:15.175515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:15.175476    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-d2m4k" podStartSLOduration=2.175465059 podStartE2EDuration="2.175465059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:15.174552286 +0000 UTC m=+253.224583470" watchObservedRunningTime="2026-11-03 14:16:15.175465059 +0000 UTC m=+253.225496241"
Nov 03 14:16:16.170905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:16.170874    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:18.290407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:18.290367    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:18.504259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:18.504218    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:18.846976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:18.846939    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:19.069012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:19.068984    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:19.453668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:19.453642    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:19.688896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:19.688864    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:20.058830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:20.058801    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:20.294068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:20.294037    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:26.073717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:26.073689    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-867995fc99-zzfz8"
Nov 03 14:16:27.983841 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:16:27.983809    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:16:29.453927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:29.453892    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:29.700149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:29.700113    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:32.461650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:32.461621    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:32.693187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:32.693157    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:35.060805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:35.060758    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:35.281353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:35.281307    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:37.897993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:37.897960    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:38.132311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.132277    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:38.349283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.349249    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:38.550062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.550034    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"]
Nov 03 14:16:38.557070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.557045    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.577291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.577269    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 14:16:38.578940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.578916    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 14:16:38.580672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.580649    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"]
Nov 03 14:16:38.592013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.591994    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-x4mhn\""
Nov 03 14:16:38.662862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.662839    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:38.663196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.663181    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29b46fcd-3054-4c92-ab92-d4811de95f03-tmp\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.663269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.663220    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7khg\" (UniqueName: \"kubernetes.io/projected/29b46fcd-3054-4c92-ab92-d4811de95f03-kube-api-access-b7khg\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.663346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.663328    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b46fcd-3054-4c92-ab92-d4811de95f03-tls-certs\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.763414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.763388    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"]
Nov 03 14:16:38.764600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.764574    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29b46fcd-3054-4c92-ab92-d4811de95f03-tmp\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.764729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.764638    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7khg\" (UniqueName: \"kubernetes.io/projected/29b46fcd-3054-4c92-ab92-d4811de95f03-kube-api-access-b7khg\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.764729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.764674    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b46fcd-3054-4c92-ab92-d4811de95f03-tls-certs\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.767000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.766484    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.767563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.767540    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/29b46fcd-3054-4c92-ab92-d4811de95f03-tmp\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.767730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.767706    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29b46fcd-3054-4c92-ab92-d4811de95f03-tls-certs\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.786278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.786220    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7khg\" (UniqueName: \"kubernetes.io/projected/29b46fcd-3054-4c92-ab92-d4811de95f03-kube-api-access-b7khg\") pod \"kube-auth-proxy-76667dfcdc-6cmzw\" (UID: \"29b46fcd-3054-4c92-ab92-d4811de95f03\") " pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.793287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.793211    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"]
Nov 03 14:16:38.805640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.805618    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-tn222\""
Nov 03 14:16:38.865667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.865637    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmg8\" (UniqueName: \"kubernetes.io/projected/2dfe3d02-9865-458c-9d3f-9b822be998dd-kube-api-access-6vmg8\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.865824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.865671    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.865824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.865692    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.865954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.865890    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfe3d02-9865-458c-9d3f-9b822be998dd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.865954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.865939    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.866042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.866023    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.866082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.866048    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.866165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.866098    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.866165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.866115    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.869593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.869571    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"
Nov 03 14:16:38.938881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.938852    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:38.966776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966751    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.966904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966785    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.966904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966849    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.966904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966880    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966911    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmg8\" (UniqueName: \"kubernetes.io/projected/2dfe3d02-9865-458c-9d3f-9b822be998dd-kube-api-access-6vmg8\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966945    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.966970    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967073    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfe3d02-9865-458c-9d3f-9b822be998dd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967108    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967279    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967325    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967565    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967615    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-data\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.967799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.967780    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dfe3d02-9865-458c-9d3f-9b822be998dd-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.969576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.969554    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.969800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.969773    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.989804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.989776    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dfe3d02-9865-458c-9d3f-9b822be998dd-istio-token\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:38.989916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:38.989822    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmg8\" (UniqueName: \"kubernetes.io/projected/2dfe3d02-9865-458c-9d3f-9b822be998dd-kube-api-access-6vmg8\") pod \"data-science-gateway-data-science-gateway-class-85d774877-jtx2p\" (UID: \"2dfe3d02-9865-458c-9d3f-9b822be998dd\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:39.104802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.104754    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:39.158780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.158748    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:39.170046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.169351    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw"]
Nov 03 14:16:39.174537 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:39.174493    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b46fcd_3054_4c92_ab92_d4811de95f03.slice/crio-e5c5a50428e148e6da3c0051d7874d6bb25cf92e530b45d9d66becb1d95f6d16 WatchSource:0}: Error finding container e5c5a50428e148e6da3c0051d7874d6bb25cf92e530b45d9d66becb1d95f6d16: Status 404 returned error can't find the container with id e5c5a50428e148e6da3c0051d7874d6bb25cf92e530b45d9d66becb1d95f6d16
Nov 03 14:16:39.415633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.415609    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"]
Nov 03 14:16:39.417814 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:39.417791    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfe3d02_9865_458c_9d3f_9b822be998dd.slice/crio-9c1bc230d92254012798b88ce1785dcc73235c20148ec3788034a5f4baca321f WatchSource:0}: Error finding container 9c1bc230d92254012798b88ce1785dcc73235c20148ec3788034a5f4baca321f: Status 404 returned error can't find the container with id 9c1bc230d92254012798b88ce1785dcc73235c20148ec3788034a5f4baca321f
Nov 03 14:16:39.420023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.419994    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:16:39.420120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.420078    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:16:39.420120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.420107    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:16:39.897406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:39.897286    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:40.132250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.132193    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:40.183519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.183482    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p" event={"ID":"2dfe3d02-9865-458c-9d3f-9b822be998dd","Type":"ContainerStarted","Data":"e21e72259be65e7bcf11f9a5365e07d508c9317196ad3590d1692348f8b3bc83"}
Nov 03 14:16:40.183706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.183546    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p" event={"ID":"2dfe3d02-9865-458c-9d3f-9b822be998dd","Type":"ContainerStarted","Data":"9c1bc230d92254012798b88ce1785dcc73235c20148ec3788034a5f4baca321f"}
Nov 03 14:16:40.185993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.185857    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw" event={"ID":"29b46fcd-3054-4c92-ab92-d4811de95f03","Type":"ContainerStarted","Data":"e5c5a50428e148e6da3c0051d7874d6bb25cf92e530b45d9d66becb1d95f6d16"}
Nov 03 14:16:40.245735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.245034    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p" podStartSLOduration=2.245020689 podStartE2EDuration="2.245020689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:40.244350404 +0000 UTC m=+278.294381588" watchObservedRunningTime="2026-11-03 14:16:40.245020689 +0000 UTC m=+278.295051868"
Nov 03 14:16:40.349033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.348817    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:40.863102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:40.863070    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:41.096951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.096919    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:41.106040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.106010    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:41.111937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.111914    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:41.193874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.193845    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:41.195607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.195589    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-85d774877-jtx2p"
Nov 03 14:16:41.310279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.310248    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:41.529190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.529117    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:41.784712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:41.784637    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:42.197998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:42.197959    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw" event={"ID":"29b46fcd-3054-4c92-ab92-d4811de95f03","Type":"ContainerStarted","Data":"29105b725998c053ccf4e3f63a147a9e5081077122a8b769fe7883404e5e272d"}
Nov 03 14:16:42.201797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:42.201770    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:42.246502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:42.246456    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-76667dfcdc-6cmzw" podStartSLOduration=1.88311823 podStartE2EDuration="4.246441615s" totalImagesPullingTime="2.363323385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:42.245302971 +0000 UTC m=+280.295334155" watchObservedRunningTime="2026-11-03 14:16:42.246441615 +0000 UTC m=+280.296472798"
Nov 03 14:16:42.421701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:42.421671    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:48.461130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:48.461036    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:48.682998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:48.682970    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:49.338136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.338105    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cllp7"]
Nov 03 14:16:49.343258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.343224    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:49.357545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.357521    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xxk\" (UniqueName: \"kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk\") pod \"authorino-57f7f8659f-cllp7\" (UID: \"39628c7e-a9f9-4384-bef4-9251aff90115\") " pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:49.361366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.361342    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cllp7"]
Nov 03 14:16:49.372017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.371973    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l7tn5\""
Nov 03 14:16:49.372017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.371989    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 14:16:49.372192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.372083    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 14:16:49.458355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.458320    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xxk\" (UniqueName: \"kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk\") pod \"authorino-57f7f8659f-cllp7\" (UID: \"39628c7e-a9f9-4384-bef4-9251aff90115\") " pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:49.480195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.480171    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xxk\" (UniqueName: \"kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk\") pod \"authorino-57f7f8659f-cllp7\" (UID: \"39628c7e-a9f9-4384-bef4-9251aff90115\") " pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:49.515697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.515673    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:49.653210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.653136    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:49.731911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.731873    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:49.943882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:49.943860    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cllp7"]
Nov 03 14:16:49.945859 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:49.945832    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39628c7e_a9f9_4384_bef4_9251aff90115.slice/crio-cddda1ba966b211c01b68005f23680cdf65f0f943df34e53f7e6b21a0734ff0f WatchSource:0}: Error finding container cddda1ba966b211c01b68005f23680cdf65f0f943df34e53f7e6b21a0734ff0f: Status 404 returned error can't find the container with id cddda1ba966b211c01b68005f23680cdf65f0f943df34e53f7e6b21a0734ff0f
Nov 03 14:16:50.019785 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.019758    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xvz6f"]
Nov 03 14:16:50.022690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.022674    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:16:50.027008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.026986    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:16:50.029651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.029632    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.045069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.045045    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xvz6f"]
Nov 03 14:16:50.047553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.047534    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 14:16:50.047784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.047767    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4qrks\""
Nov 03 14:16:50.057561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.057542    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:16:50.064330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.064296    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jbv\" (UniqueName: \"kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv\") pod \"authorino-56c6c9ddd5-xvz6f\" (UID: \"527db03e-3cb6-4e62-b21b-7762de893314\") " pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:16:50.064434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.064413    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvdt\" (UniqueName: \"kubernetes.io/projected/5135bcc0-45a6-414e-910c-f68bed8be41a-kube-api-access-9tvdt\") pod \"limitador-limitador-7c9b87d574-f8vvn\" (UID: \"5135bcc0-45a6-414e-910c-f68bed8be41a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.064495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.064478    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5135bcc0-45a6-414e-910c-f68bed8be41a-config-file\") pod \"limitador-limitador-7c9b87d574-f8vvn\" (UID: \"5135bcc0-45a6-414e-910c-f68bed8be41a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.098972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.098942    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:16:50.165336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.165298    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jbv\" (UniqueName: \"kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv\") pod \"authorino-56c6c9ddd5-xvz6f\" (UID: \"527db03e-3cb6-4e62-b21b-7762de893314\") " pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:16:50.165469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.165446    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvdt\" (UniqueName: \"kubernetes.io/projected/5135bcc0-45a6-414e-910c-f68bed8be41a-kube-api-access-9tvdt\") pod \"limitador-limitador-7c9b87d574-f8vvn\" (UID: \"5135bcc0-45a6-414e-910c-f68bed8be41a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.165542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.165524    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5135bcc0-45a6-414e-910c-f68bed8be41a-config-file\") pod \"limitador-limitador-7c9b87d574-f8vvn\" (UID: \"5135bcc0-45a6-414e-910c-f68bed8be41a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.166102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.166084    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5135bcc0-45a6-414e-910c-f68bed8be41a-config-file\") pod \"limitador-limitador-7c9b87d574-f8vvn\" (UID: \"5135bcc0-45a6-414e-910c-f68bed8be41a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.187094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.187065    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jbv\" (UniqueName: \"kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv\") pod \"authorino-56c6c9ddd5-xvz6f\" (UID: \"527db03e-3cb6-4e62-b21b-7762de893314\") " pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:16:50.187175 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.187093    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvdt\" (UniqueName: \"kubernetes.io/projected/5135bcc0-45a6-414e-910c-f68bed8be41a-kube-api-access-9tvdt\") pod \"limitador-limitador-7c9b87d574-f8vvn\" (UID: \"5135bcc0-45a6-414e-910c-f68bed8be41a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.232509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.232448    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cllp7" event={"ID":"39628c7e-a9f9-4384-bef4-9251aff90115","Type":"ContainerStarted","Data":"cddda1ba966b211c01b68005f23680cdf65f0f943df34e53f7e6b21a0734ff0f"}
Nov 03 14:16:50.334698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.334674    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:16:50.341586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.341565    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:50.628687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.628665    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xvz6f"]
Nov 03 14:16:50.630374 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:50.630345    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod527db03e_3cb6_4e62_b21b_7762de893314.slice/crio-7f5e9fcbdc163e5e9bdd83e985dea15c45239955d4cc97da6f58f0f0e74faf76 WatchSource:0}: Error finding container 7f5e9fcbdc163e5e9bdd83e985dea15c45239955d4cc97da6f58f0f0e74faf76: Status 404 returned error can't find the container with id 7f5e9fcbdc163e5e9bdd83e985dea15c45239955d4cc97da6f58f0f0e74faf76
Nov 03 14:16:50.641336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:50.641311    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:16:50.643757 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:16:50.643733    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5135bcc0_45a6_414e_910c_f68bed8be41a.slice/crio-5ca6c12293add3453edc4792a9d5ce96fa8ce25311dbace7b61499ac25aa398d WatchSource:0}: Error finding container 5ca6c12293add3453edc4792a9d5ce96fa8ce25311dbace7b61499ac25aa398d: Status 404 returned error can't find the container with id 5ca6c12293add3453edc4792a9d5ce96fa8ce25311dbace7b61499ac25aa398d
Nov 03 14:16:51.252899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:51.252801    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f" event={"ID":"527db03e-3cb6-4e62-b21b-7762de893314","Type":"ContainerStarted","Data":"7f5e9fcbdc163e5e9bdd83e985dea15c45239955d4cc97da6f58f0f0e74faf76"}
Nov 03 14:16:51.257487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:51.257370    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn" event={"ID":"5135bcc0-45a6-414e-910c-f68bed8be41a","Type":"ContainerStarted","Data":"5ca6c12293add3453edc4792a9d5ce96fa8ce25311dbace7b61499ac25aa398d"}
Nov 03 14:16:52.685634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:52.685598    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:52.938904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:52.938811    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:54.275941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.275902    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn" event={"ID":"5135bcc0-45a6-414e-910c-f68bed8be41a","Type":"ContainerStarted","Data":"1c90ebf3d5ed4c1ada14107be912cc0828f73f3cd12933a745a648901bcf5af0"}
Nov 03 14:16:54.276380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.276073    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:16:54.277385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.277346    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cllp7" event={"ID":"39628c7e-a9f9-4384-bef4-9251aff90115","Type":"ContainerStarted","Data":"753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261"}
Nov 03 14:16:54.278707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.278684    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f" event={"ID":"527db03e-3cb6-4e62-b21b-7762de893314","Type":"ContainerStarted","Data":"6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b"}
Nov 03 14:16:54.322486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.322442    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn" podStartSLOduration=2.49685074 podStartE2EDuration="5.322430233s" totalImagesPullingTime="2.825579493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:54.321201841 +0000 UTC m=+292.371233025" watchObservedRunningTime="2026-11-03 14:16:54.322430233 +0000 UTC m=+292.372461415"
Nov 03 14:16:54.367790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.367747    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f" podStartSLOduration=2.348878442 podStartE2EDuration="5.367735251s" totalImagesPullingTime="3.018856809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:54.367101425 +0000 UTC m=+292.417132609" watchObservedRunningTime="2026-11-03 14:16:54.367735251 +0000 UTC m=+292.417766458"
Nov 03 14:16:54.399189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.399167    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cllp7"]
Nov 03 14:16:54.420951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:54.420900    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-cllp7" podStartSLOduration=1.986668963 podStartE2EDuration="5.420886477s" totalImagesPullingTime="3.434217514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:16:54.419794611 +0000 UTC m=+292.469825795" watchObservedRunningTime="2026-11-03 14:16:54.420886477 +0000 UTC m=+292.470917660"
Nov 03 14:16:55.986453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:55.986420    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:16:56.287480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:56.287371    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-cllp7" podUID="39628c7e-a9f9-4384-bef4-9251aff90115" containerName="authorino" containerID="cri-o://753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261" gracePeriod=30
Nov 03 14:16:56.535013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:56.534990    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:56.622607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:56.622540    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk\" (UniqueName: \"kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk\") pod \"39628c7e-a9f9-4384-bef4-9251aff90115\" (UID: \"39628c7e-a9f9-4384-bef4-9251aff90115\") "
Nov 03 14:16:56.624642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:56.624615    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk" pod "39628c7e-a9f9-4384-bef4-9251aff90115" (UID: "39628c7e-a9f9-4384-bef4-9251aff90115"). InnerVolumeSpecName "kube-api-access-j4xxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:16:56.724159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:56.724131    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xxk\" (UniqueName: \"kubernetes.io/projected/39628c7e-a9f9-4384-bef4-9251aff90115-kube-api-access-j4xxk\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:16:57.292696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.292656    6481 generic.go:356] "Generic (PLEG): container finished" podID="39628c7e-a9f9-4384-bef4-9251aff90115" containerID="753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261" exitCode=0
Nov 03 14:16:57.293184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.292720    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-cllp7"
Nov 03 14:16:57.293184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.292747    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cllp7" event={"ID":"39628c7e-a9f9-4384-bef4-9251aff90115","Type":"ContainerDied","Data":"753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261"}
Nov 03 14:16:57.293184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.292794    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-cllp7" event={"ID":"39628c7e-a9f9-4384-bef4-9251aff90115","Type":"ContainerDied","Data":"cddda1ba966b211c01b68005f23680cdf65f0f943df34e53f7e6b21a0734ff0f"}
Nov 03 14:16:57.293184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.292814    6481 scope.go:122] "RemoveContainer" containerID="753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261"
Nov 03 14:16:57.302345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.302326    6481 scope.go:122] "RemoveContainer" containerID="753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261"
Nov 03 14:16:57.302648 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:16:57.302626    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261\": container with ID starting with 753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261 not found: ID does not exist" containerID="753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261"
Nov 03 14:16:57.302748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.302652    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261"} err="failed to get container status \"753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261\": rpc error: code = NotFound desc = could not find container \"753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261\": container with ID starting with 753754029afb102bbf3fddac671a0c9c6f6b0f39f1cc0a487050048c177cc261 not found: ID does not exist"
Nov 03 14:16:57.348471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.348441    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cllp7"]
Nov 03 14:16:57.355013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:57.354988    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-cllp7"]
Nov 03 14:16:57.984789 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:16:57.984757    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:16:58.065532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:16:58.065500    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39628c7e-a9f9-4384-bef4-9251aff90115" path="/var/lib/kubelet/pods/39628c7e-a9f9-4384-bef4-9251aff90115/volumes"
Nov 03 14:17:05.284132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:05.284099    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"
Nov 03 14:17:10.665280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:10.665242    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:10.892656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:10.892620    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:11.501622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:11.501587    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:27.985056 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:17:27.985019    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:17:31.162869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:31.162828    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:31.398925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:31.398891    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:31.639972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:31.639891    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:35.079135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:35.079103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:35.309930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:35.309901    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:38.888241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:38.888200    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:39.120788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.120752    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:39.204983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.204950    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-78fsn"]
Nov 03 14:17:39.205332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.205319    6481 cpu_manager.go:414] "containerMap: removing container" podUID="39628c7e-a9f9-4384-bef4-9251aff90115" containerName="authorino"
Nov 03 14:17:39.205385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.205334    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39628c7e-a9f9-4384-bef4-9251aff90115" containerName="authorino"
Nov 03 14:17:39.205422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.205395    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="39628c7e-a9f9-4384-bef4-9251aff90115" containerName="authorino"
Nov 03 14:17:39.211613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.211593    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.228904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.228884    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-78fsn"]
Nov 03 14:17:39.246696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.246669    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-nmtcb"]
Nov 03 14:17:39.249025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.249010    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.252597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.252574    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 14:17:39.252836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.252819    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 14:17:39.252836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.252834    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-86mgp\""
Nov 03 14:17:39.252932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.252826    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 14:17:39.271331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.271308    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-nmtcb"]
Nov 03 14:17:39.287053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.287033    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07e51ae4-b9c2-4424-b835-81b5b8f33934-cert\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.287158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.287059    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxz6c\" (UniqueName: \"kubernetes.io/projected/07e51ae4-b9c2-4424-b835-81b5b8f33934-kube-api-access-nxz6c\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.287158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.287082    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e51ae4-b9c2-4424-b835-81b5b8f33934-metrics-certs\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.287158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.287104    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e1d2c0b-5871-481c-83b2-48f62884699d-tls-certs\") pod \"model-serving-api-75bd89fdfc-78fsn\" (UID: \"5e1d2c0b-5871-481c-83b2-48f62884699d\") " pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.287292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.287278    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjmg\" (UniqueName: \"kubernetes.io/projected/5e1d2c0b-5871-481c-83b2-48f62884699d-kube-api-access-6mjmg\") pod \"model-serving-api-75bd89fdfc-78fsn\" (UID: \"5e1d2c0b-5871-481c-83b2-48f62884699d\") " pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.341869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.341843    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:39.353642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.353622    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 14:17:39.353737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.353715    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 14:17:39.353815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.353790    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-fcr5j\""
Nov 03 14:17:39.388029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.388005    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjmg\" (UniqueName: \"kubernetes.io/projected/5e1d2c0b-5871-481c-83b2-48f62884699d-kube-api-access-6mjmg\") pod \"model-serving-api-75bd89fdfc-78fsn\" (UID: \"5e1d2c0b-5871-481c-83b2-48f62884699d\") " pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.388131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.388075    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07e51ae4-b9c2-4424-b835-81b5b8f33934-cert\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.388131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.388091    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxz6c\" (UniqueName: \"kubernetes.io/projected/07e51ae4-b9c2-4424-b835-81b5b8f33934-kube-api-access-nxz6c\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.388131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.388110    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e51ae4-b9c2-4424-b835-81b5b8f33934-metrics-certs\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.388131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.388132    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e1d2c0b-5871-481c-83b2-48f62884699d-tls-certs\") pod \"model-serving-api-75bd89fdfc-78fsn\" (UID: \"5e1d2c0b-5871-481c-83b2-48f62884699d\") " pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.390725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.390705    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07e51ae4-b9c2-4424-b835-81b5b8f33934-metrics-certs\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.390808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.390705    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e1d2c0b-5871-481c-83b2-48f62884699d-tls-certs\") pod \"model-serving-api-75bd89fdfc-78fsn\" (UID: \"5e1d2c0b-5871-481c-83b2-48f62884699d\") " pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.390846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.390813    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07e51ae4-b9c2-4424-b835-81b5b8f33934-cert\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.410755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.410730    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxz6c\" (UniqueName: \"kubernetes.io/projected/07e51ae4-b9c2-4424-b835-81b5b8f33934-kube-api-access-nxz6c\") pod \"odh-model-controller-769875db56-nmtcb\" (UID: \"07e51ae4-b9c2-4424-b835-81b5b8f33934\") " pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.410755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.410748    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjmg\" (UniqueName: \"kubernetes.io/projected/5e1d2c0b-5871-481c-83b2-48f62884699d-kube-api-access-6mjmg\") pod \"model-serving-api-75bd89fdfc-78fsn\" (UID: \"5e1d2c0b-5871-481c-83b2-48f62884699d\") " pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.522407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.522331    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:39.558663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.558621    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:39.560539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.560513    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:39.779862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.779799    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:39.812047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.812021    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"]
Nov 03 14:17:39.814989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.814967    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.819154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.819130    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-78fsn"]
Nov 03 14:17:39.823323 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:17:39.823170    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1d2c0b_5871_481c_83b2_48f62884699d.slice/crio-28c1a86de4101075725d28aae4397d6b4359b1149c7303d39ae692e232a5d561 WatchSource:0}: Error finding container 28c1a86de4101075725d28aae4397d6b4359b1149c7303d39ae692e232a5d561: Status 404 returned error can't find the container with id 28c1a86de4101075725d28aae4397d6b4359b1149c7303d39ae692e232a5d561
Nov 03 14:17:39.837606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.837586    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-cxnt4\""
Nov 03 14:17:39.837725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.837622    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 14:17:39.837725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.837697    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 14:17:39.837863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.837810    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 14:17:39.838797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.838770    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"]
Nov 03 14:17:39.868206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.868162    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-nmtcb"]
Nov 03 14:17:39.870316 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:17:39.870294    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e51ae4_b9c2_4424_b835_81b5b8f33934.slice/crio-a7fd2dc7bc4987b4ca6d4257d40f3f44c1d6dc14cdd959110b3aca6f3e4383aa WatchSource:0}: Error finding container a7fd2dc7bc4987b4ca6d4257d40f3f44c1d6dc14cdd959110b3aca6f3e4383aa: Status 404 returned error can't find the container with id a7fd2dc7bc4987b4ca6d4257d40f3f44c1d6dc14cdd959110b3aca6f3e4383aa
Nov 03 14:17:39.892075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.892047    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e5dafb2a-ee70-4307-8977-f1289ea765aa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.892412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.892108    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dafb2a-ee70-4307-8977-f1289ea765aa-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.892412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.892192    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5dafb2a-ee70-4307-8977-f1289ea765aa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.892412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.892223    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bh4\" (UniqueName: \"kubernetes.io/projected/e5dafb2a-ee70-4307-8977-f1289ea765aa-kube-api-access-q2bh4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.993349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.993318    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e5dafb2a-ee70-4307-8977-f1289ea765aa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.993494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.993374    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dafb2a-ee70-4307-8977-f1289ea765aa-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.993494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.993419    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5dafb2a-ee70-4307-8977-f1289ea765aa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.993494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.993445    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bh4\" (UniqueName: \"kubernetes.io/projected/e5dafb2a-ee70-4307-8977-f1289ea765aa-kube-api-access-q2bh4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.994155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.994134    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dafb2a-ee70-4307-8977-f1289ea765aa-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.994329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.994309    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/e5dafb2a-ee70-4307-8977-f1289ea765aa-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:39.995941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:39.995920    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e5dafb2a-ee70-4307-8977-f1289ea765aa-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:40.016559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.016528    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:40.016668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.016580    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bh4\" (UniqueName: \"kubernetes.io/projected/e5dafb2a-ee70-4307-8977-f1289ea765aa-kube-api-access-q2bh4\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-kjghr\" (UID: \"e5dafb2a-ee70-4307-8977-f1289ea765aa\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:40.128938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.128868    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:40.233581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.233513    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:40.429434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.429402    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"]
Nov 03 14:17:40.433645 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:17:40.433613    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dafb2a_ee70_4307_8977_f1289ea765aa.slice/crio-2f7eaba6f636e98257eba024bd49733535d536e9011798df4a594f7f415aa1cb WatchSource:0}: Error finding container 2f7eaba6f636e98257eba024bd49733535d536e9011798df4a594f7f415aa1cb: Status 404 returned error can't find the container with id 2f7eaba6f636e98257eba024bd49733535d536e9011798df4a594f7f415aa1cb
Nov 03 14:17:40.483462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.483389    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr" event={"ID":"e5dafb2a-ee70-4307-8977-f1289ea765aa","Type":"ContainerStarted","Data":"2f7eaba6f636e98257eba024bd49733535d536e9011798df4a594f7f415aa1cb"}
Nov 03 14:17:40.485680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.485650    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn" event={"ID":"5e1d2c0b-5871-481c-83b2-48f62884699d","Type":"ContainerStarted","Data":"28c1a86de4101075725d28aae4397d6b4359b1149c7303d39ae692e232a5d561"}
Nov 03 14:17:40.487502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:40.487477    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-nmtcb" event={"ID":"07e51ae4-b9c2-4424-b835-81b5b8f33934","Type":"ContainerStarted","Data":"a7fd2dc7bc4987b4ca6d4257d40f3f44c1d6dc14cdd959110b3aca6f3e4383aa"}
Nov 03 14:17:41.759466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:41.759432    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:41.981697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:41.981656    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:42.205728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:42.205694    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:42.502837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:42.502753    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn" event={"ID":"5e1d2c0b-5871-481c-83b2-48f62884699d","Type":"ContainerStarted","Data":"5f5bc073aea24486722927abaab5a7b2eb49cfcfa63fe5c0bbefed0131fb5fc2"}
Nov 03 14:17:42.503654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:42.503632    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:42.548525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:42.548467    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn" podStartSLOduration=1.611495752 podStartE2EDuration="3.548447915s" totalImagesPullingTime="1.936952163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:42.54772462 +0000 UTC m=+340.597755804" watchObservedRunningTime="2026-11-03 14:17:42.548447915 +0000 UTC m=+340.598479099"
Nov 03 14:17:43.511632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:43.511283    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-nmtcb" event={"ID":"07e51ae4-b9c2-4424-b835-81b5b8f33934","Type":"ContainerStarted","Data":"1f60da3ff3dcc75f594041f59cdf51e9f272f344980b8b731fc2b47f69b422fe"}
Nov 03 14:17:43.511632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:43.511578    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:51.554803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:51.554762    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr" event={"ID":"e5dafb2a-ee70-4307-8977-f1289ea765aa","Type":"ContainerStarted","Data":"13622baead392f104ac41756a587d78cd3f3cc105a2f5623f366d89dfb8a06eb"}
Nov 03 14:17:51.555169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:51.554864    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:17:51.602355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:51.602298    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-nmtcb" podStartSLOduration=9.137895651000001 podStartE2EDuration="12.602280057s" totalImagesPullingTime="3.464384406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:43.556464454 +0000 UTC m=+341.606495637" watchObservedRunningTime="2026-11-03 14:17:51.602280057 +0000 UTC m=+349.652311241"
Nov 03 14:17:51.603492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:51.603447    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr" podStartSLOduration=1.640988779 podStartE2EDuration="12.603430856s" totalImagesPullingTime="10.962442077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:51.601537774 +0000 UTC m=+349.651568958" watchObservedRunningTime="2026-11-03 14:17:51.603430856 +0000 UTC m=+349.653462040"
Nov 03 14:17:54.520142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:54.520109    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-nmtcb"
Nov 03 14:17:54.522043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:54.522018    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-78fsn"
Nov 03 14:17:56.136098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:17:56.136064    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:17:57.984767 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:17:57.984731    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:18:02.566558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:02.566515    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-kjghr"
Nov 03 14:18:08.620821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:08.620789    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:08.841544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:08.841507    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:10.064048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:10.064015    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:10.282040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:10.282008    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:12.385605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:12.385570    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:18:19.720216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:19.720181    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:19.885297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:19.885253    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrwvj"]
Nov 03 14:18:19.942247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:19.942204    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:20.136439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.136296    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrwvj"]
Nov 03 14:18:20.136439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.136343    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:20.143407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.143378    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrwvj"]
Nov 03 14:18:20.143558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.143428    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2p2jc"]
Nov 03 14:18:20.166416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.166395    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.166529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.166395    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2p2jc"]
Nov 03 14:18:20.172609 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:18:20.172582    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dsd2c], unattached volumes=[], failed to process volumes=[kube-api-access-dsd2c]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-lrwvj" podUID="17ea0b11-f070-4777-83aa-af37d4a6b216"
Nov 03 14:18:20.179808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.179789    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert\") pod \"authorino-6889cbd8c4-2p2jc\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") " pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.179898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.179829    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhzkp\" (UniqueName: \"kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp\") pod \"authorino-6889cbd8c4-2p2jc\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") " pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.179943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.179907    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsd2c\" (UniqueName: \"kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c\") pod \"authorino-6f6944bcdd-lrwvj\" (UID: \"17ea0b11-f070-4777-83aa-af37d4a6b216\") " pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:20.190458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.190438    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 14:18:20.255015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.254980    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2p2jc"]
Nov 03 14:18:20.255222 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:18:20.255202    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dhzkp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-2p2jc" podUID="82dd4a97-0764-4322-af0b-7e4bc4f678fa"
Nov 03 14:18:20.280742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.280707    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert\") pod \"authorino-6889cbd8c4-2p2jc\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") " pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.280873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.280753    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhzkp\" (UniqueName: \"kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp\") pod \"authorino-6889cbd8c4-2p2jc\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") " pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.280873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.280795    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsd2c\" (UniqueName: \"kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c\") pod \"authorino-6f6944bcdd-lrwvj\" (UID: \"17ea0b11-f070-4777-83aa-af37d4a6b216\") " pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:20.282073 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.282049    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5ffb77fcd4-8kmbf"]
Nov 03 14:18:20.283511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.283479    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert\") pod \"authorino-6889cbd8c4-2p2jc\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") " pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.302655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.302635    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhzkp\" (UniqueName: \"kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp\") pod \"authorino-6889cbd8c4-2p2jc\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") " pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.303069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.303024    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsd2c\" (UniqueName: \"kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c\") pod \"authorino-6f6944bcdd-lrwvj\" (UID: \"17ea0b11-f070-4777-83aa-af37d4a6b216\") " pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:20.304517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.304495    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ffb77fcd4-8kmbf"]
Nov 03 14:18:20.304614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.304599    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.382331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.382298    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert\") pod \"authorino-5ffb77fcd4-8kmbf\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") " pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.382537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.382484    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rnw\" (UniqueName: \"kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw\") pod \"authorino-5ffb77fcd4-8kmbf\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") " pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.483702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.483655    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert\") pod \"authorino-5ffb77fcd4-8kmbf\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") " pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.483878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.483834    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rnw\" (UniqueName: \"kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw\") pod \"authorino-5ffb77fcd4-8kmbf\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") " pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.486293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.486266    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert\") pod \"authorino-5ffb77fcd4-8kmbf\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") " pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.506652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.506623    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rnw\" (UniqueName: \"kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw\") pod \"authorino-5ffb77fcd4-8kmbf\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") " pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.646911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.646876    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:18:20.698010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.697977    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:20.698176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.697977    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.726331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.725379    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:20.739695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.739379    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:20.787129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.787105    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp\" (UniqueName: \"kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp\") pod \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") "
Nov 03 14:18:20.787289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.787201    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert\" (UniqueName: \"kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert\") pod \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\" (UID: \"82dd4a97-0764-4322-af0b-7e4bc4f678fa\") "
Nov 03 14:18:20.787359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.787294    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c\" (UniqueName: \"kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c\") pod \"17ea0b11-f070-4777-83aa-af37d4a6b216\" (UID: \"17ea0b11-f070-4777-83aa-af37d4a6b216\") "
Nov 03 14:18:20.789251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.789203    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert" pod "82dd4a97-0764-4322-af0b-7e4bc4f678fa" (UID: "82dd4a97-0764-4322-af0b-7e4bc4f678fa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:18:20.789335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.789222    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c" pod "17ea0b11-f070-4777-83aa-af37d4a6b216" (UID: "17ea0b11-f070-4777-83aa-af37d4a6b216"). InnerVolumeSpecName "kube-api-access-dsd2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:18:20.789335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.789261    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp" pod "82dd4a97-0764-4322-af0b-7e4bc4f678fa" (UID: "82dd4a97-0764-4322-af0b-7e4bc4f678fa"). InnerVolumeSpecName "kube-api-access-dhzkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:18:20.887860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.887823    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsd2c\" (UniqueName: \"kubernetes.io/projected/17ea0b11-f070-4777-83aa-af37d4a6b216-kube-api-access-dsd2c\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:18:20.887860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.887854    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhzkp\" (UniqueName: \"kubernetes.io/projected/82dd4a97-0764-4322-af0b-7e4bc4f678fa-kube-api-access-dhzkp\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:18:20.888032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.887868    6481 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/82dd4a97-0764-4322-af0b-7e4bc4f678fa-tls-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:18:20.956945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:20.956844    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5ffb77fcd4-8kmbf"]
Nov 03 14:18:20.959205 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:18:20.959178    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5174748c_e439_4268_9ed8_92318a988c71.slice/crio-f8280433e01061e92c01d3a6c0874d8514e602ad8d2a544aeee7ab1dd560d047 WatchSource:0}: Error finding container f8280433e01061e92c01d3a6c0874d8514e602ad8d2a544aeee7ab1dd560d047: Status 404 returned error can't find the container with id f8280433e01061e92c01d3a6c0874d8514e602ad8d2a544aeee7ab1dd560d047
Nov 03 14:18:21.704580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.704544    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf" event={"ID":"5174748c-e439-4268-9ed8-92318a988c71","Type":"ContainerStarted","Data":"f8280433e01061e92c01d3a6c0874d8514e602ad8d2a544aeee7ab1dd560d047"}
Nov 03 14:18:21.704580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.704570    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-2p2jc"
Nov 03 14:18:21.704780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.704742    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lrwvj"
Nov 03 14:18:21.789925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.789887    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrwvj"]
Nov 03 14:18:21.797798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.797775    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lrwvj"]
Nov 03 14:18:21.861703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.861675    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2p2jc"]
Nov 03 14:18:21.869362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:21.869337    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-2p2jc"]
Nov 03 14:18:22.065440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.065373    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ea0b11-f070-4777-83aa-af37d4a6b216" path="/var/lib/kubelet/pods/17ea0b11-f070-4777-83aa-af37d4a6b216/volumes"
Nov 03 14:18:22.065633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.065621    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dd4a97-0764-4322-af0b-7e4bc4f678fa" path="/var/lib/kubelet/pods/82dd4a97-0764-4322-af0b-7e4bc4f678fa/volumes"
Nov 03 14:18:22.711459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.711422    6481 generic.go:356] "Generic (PLEG): container finished" podID="053a3628-1b59-4617-801f-21d29607180b" containerID="becef13b733565201b43de0616646848747ca94c2390ebb7f29f56b270d093d4" exitCode=0
Nov 03 14:18:22.711682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.711512    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" event={"ID":"053a3628-1b59-4617-801f-21d29607180b","Type":"ContainerDied","Data":"becef13b733565201b43de0616646848747ca94c2390ebb7f29f56b270d093d4"}
Nov 03 14:18:22.712003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.711977    6481 scope.go:122] "RemoveContainer" containerID="becef13b733565201b43de0616646848747ca94c2390ebb7f29f56b270d093d4"
Nov 03 14:18:22.713318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.713285    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:18:22.713416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.713374    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:18:22.713416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.713398    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf" event={"ID":"5174748c-e439-4268-9ed8-92318a988c71","Type":"ContainerStarted","Data":"033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470"}
Nov 03 14:18:22.713534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.713417    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:18:22.830632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.830513    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:18:22.833830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.833752    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf" podStartSLOduration=1.613877836 podStartE2EDuration="2.833733764s" totalImagesPullingTime="1.219855928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:18:22.83336033 +0000 UTC m=+380.883391510" watchObservedRunningTime="2026-11-03 14:18:22.833733764 +0000 UTC m=+380.883764949"
Nov 03 14:18:22.882282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.879840    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xvz6f"]
Nov 03 14:18:22.882476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:22.882293    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f" podUID="527db03e-3cb6-4e62-b21b-7762de893314" containerName="authorino" containerID="cri-o://6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b" gracePeriod=30
Nov 03 14:18:23.194292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.194192    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:18:23.222529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.221609    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv\" (UniqueName: \"kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv\") pod \"527db03e-3cb6-4e62-b21b-7762de893314\" (UID: \"527db03e-3cb6-4e62-b21b-7762de893314\") "
Nov 03 14:18:23.227617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.227577    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv" pod "527db03e-3cb6-4e62-b21b-7762de893314" (UID: "527db03e-3cb6-4e62-b21b-7762de893314"). InnerVolumeSpecName "kube-api-access-x2jbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:18:23.325642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.325607    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jbv\" (UniqueName: \"kubernetes.io/projected/527db03e-3cb6-4e62-b21b-7762de893314-kube-api-access-x2jbv\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:18:23.730580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.730541    6481 generic.go:356] "Generic (PLEG): container finished" podID="527db03e-3cb6-4e62-b21b-7762de893314" containerID="6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b" exitCode=0
Nov 03 14:18:23.730580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.730581    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f"
Nov 03 14:18:23.730839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.730631    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f" event={"ID":"527db03e-3cb6-4e62-b21b-7762de893314","Type":"ContainerDied","Data":"6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b"}
Nov 03 14:18:23.730839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.730677    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xvz6f" event={"ID":"527db03e-3cb6-4e62-b21b-7762de893314","Type":"ContainerDied","Data":"7f5e9fcbdc163e5e9bdd83e985dea15c45239955d4cc97da6f58f0f0e74faf76"}
Nov 03 14:18:23.730839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.730701    6481 scope.go:122] "RemoveContainer" containerID="6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b"
Nov 03 14:18:23.732957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.732935    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" event={"ID":"053a3628-1b59-4617-801f-21d29607180b","Type":"ContainerStarted","Data":"17c0a9b8cfedf0cacc5a22d1225bdaa53cc37343febce1720e60678dbfbdf514"}
Nov 03 14:18:23.740409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.740343    6481 scope.go:122] "RemoveContainer" containerID="6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b"
Nov 03 14:18:23.740612 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:18:23.740590    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b\": container with ID starting with 6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b not found: ID does not exist" containerID="6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b"
Nov 03 14:18:23.740696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.740618    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b"} err="failed to get container status \"6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b\": rpc error: code = NotFound desc = could not find container \"6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b\": container with ID starting with 6c7b86cc3a0fbed32a26e54622424752e1b4002cea167d5bbffb8266ff9e056b not found: ID does not exist"
Nov 03 14:18:23.766825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.766806    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:18:23.767756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.767736    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-675z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Nov 03 14:18:23.767882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.767776    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Nov 03 14:18:23.785611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.785591    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xvz6f"]
Nov 03 14:18:23.798571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:23.798552    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xvz6f"]
Nov 03 14:18:24.067026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:24.066951    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527db03e-3cb6-4e62-b21b-7762de893314" path="/var/lib/kubelet/pods/527db03e-3cb6-4e62-b21b-7762de893314/volumes"
Nov 03 14:18:24.767098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:24.767065    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-675z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Nov 03 14:18:24.767285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:24.767117    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Nov 03 14:18:25.766870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:25.766838    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-675z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Nov 03 14:18:25.767261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:25.766891    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Nov 03 14:18:26.766753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:26.766716    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-675z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Nov 03 14:18:26.766948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:26.766777    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Nov 03 14:18:27.766802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:27.766766    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-675z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused" start-of-body=
Nov 03 14:18:27.766995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:27.766827    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.17:15021/healthz/ready\": dial tcp 10.133.0.17:15021: connect: connection refused"
Nov 03 14:18:27.985747 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:18:27.985713    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:18:28.834416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:28.834355    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:18:28.835028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:28.834964    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:18:28.835642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:28.835624    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:18:56.240219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:18:56.240160    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:18:57.984795 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:18:57.984761    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:19:17.973629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:17.973598    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:18.193383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:18.193352    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:23.013138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:23.013105    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:23.240768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:23.240735    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:27.985253 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:19:27.985204    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:19:40.030779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:40.030744    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:41.178904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:41.178873    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:41.411954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:41.411922    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:47.325523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:47.325445    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:47.829340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:47.829303    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:49.073731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:49.073701    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:49.316384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:49.316345    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:52.838534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:52.838504    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:54.319477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:54.319442    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:19:55.949176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:55.949143    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:56.336661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:56.336591    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:57.181171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.181138    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"]
Nov 03 14:19:57.182192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.181715    6481 cpu_manager.go:414] "containerMap: removing container" podUID="527db03e-3cb6-4e62-b21b-7762de893314" containerName="authorino"
Nov 03 14:19:57.182192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.181735    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527db03e-3cb6-4e62-b21b-7762de893314" containerName="authorino"
Nov 03 14:19:57.182192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.181847    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="527db03e-3cb6-4e62-b21b-7762de893314" containerName="authorino"
Nov 03 14:19:57.183940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.183922    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.215419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.215388    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 14:19:57.215745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.215727    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 14:19:57.215883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.215847    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dhg2d\""
Nov 03 14:19:57.216049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.216000    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 14:19:57.220818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.220795    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"]
Nov 03 14:19:57.290838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.290792    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.292141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.292112    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.292358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.292335    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctr9\" (UniqueName: \"kubernetes.io/projected/5b72d32b-da35-496d-a85e-327eac0da3f1-kube-api-access-7ctr9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.292511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.292494    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.293259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.293220    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72d32b-da35-496d-a85e-327eac0da3f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.293413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.293397    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.295194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.295164    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:19:57.360115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.360074    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:57.394943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.394910    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72d32b-da35-496d-a85e-327eac0da3f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.394947    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395068    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395116    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395140    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctr9\" (UniqueName: \"kubernetes.io/projected/5b72d32b-da35-496d-a85e-327eac0da3f1-kube-api-access-7ctr9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395193    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395569    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.395861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395838    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.396037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.395853    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.397404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.397383    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b72d32b-da35-496d-a85e-327eac0da3f1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.397903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.397886    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b72d32b-da35-496d-a85e-327eac0da3f1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.421264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.419926    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctr9\" (UniqueName: \"kubernetes.io/projected/5b72d32b-da35-496d-a85e-327eac0da3f1-kube-api-access-7ctr9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-pbhb4\" (UID: \"5b72d32b-da35-496d-a85e-327eac0da3f1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.505622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.505537    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:57.818979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:57.818775    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"]
Nov 03 14:19:57.821676 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:19:57.821651    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b72d32b_da35_496d_a85e_327eac0da3f1.slice/crio-36c151d456c70023f3555d39d4f3f1407d23dd1b881aa87866e1fd5705f86659 WatchSource:0}: Error finding container 36c151d456c70023f3555d39d4f3f1407d23dd1b881aa87866e1fd5705f86659: Status 404 returned error can't find the container with id 36c151d456c70023f3555d39d4f3f1407d23dd1b881aa87866e1fd5705f86659
Nov 03 14:19:57.989221 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:19:57.989182    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:19:58.159299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.159210    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4" event={"ID":"5b72d32b-da35-496d-a85e-327eac0da3f1","Type":"ContainerStarted","Data":"36c151d456c70023f3555d39d4f3f1407d23dd1b881aa87866e1fd5705f86659"}
Nov 03 14:19:58.373556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.373520    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"]
Nov 03 14:19:58.386877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.386847    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.418317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.415967    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 14:19:58.418567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.418543    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"]
Nov 03 14:19:58.507186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.507150    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.507412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.507341    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.507488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.507414    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn25x\" (UniqueName: \"kubernetes.io/projected/9c143c15-a7c2-4d1f-8e60-216aae706edf-kube-api-access-zn25x\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.507546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.507538    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c143c15-a7c2-4d1f-8e60-216aae706edf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.507643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.507624    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.507712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.507683    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.541153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.541120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:19:58.608637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.608601    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.608837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.608666    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.608837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.608740    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.608837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.608770    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.608837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.608797    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn25x\" (UniqueName: \"kubernetes.io/projected/9c143c15-a7c2-4d1f-8e60-216aae706edf-kube-api-access-zn25x\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.609047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.608841    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c143c15-a7c2-4d1f-8e60-216aae706edf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.609395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.609353    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.609610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.609426    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.609716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.609649    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.611424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.611387    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c143c15-a7c2-4d1f-8e60-216aae706edf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.612083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.612057    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c143c15-a7c2-4d1f-8e60-216aae706edf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.631992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.631971    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn25x\" (UniqueName: \"kubernetes.io/projected/9c143c15-a7c2-4d1f-8e60-216aae706edf-kube-api-access-zn25x\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tr28v\" (UID: \"9c143c15-a7c2-4d1f-8e60-216aae706edf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:58.722849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:58.722756    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:19:59.025479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:59.025453    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"]
Nov 03 14:19:59.027642 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:19:59.027614    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c143c15_a7c2_4d1f_8e60_216aae706edf.slice/crio-9f9b3aa6bf2419df680643ec6356b5d290eb0556b37cb15712d43ba6d2dc6c7f WatchSource:0}: Error finding container 9f9b3aa6bf2419df680643ec6356b5d290eb0556b37cb15712d43ba6d2dc6c7f: Status 404 returned error can't find the container with id 9f9b3aa6bf2419df680643ec6356b5d290eb0556b37cb15712d43ba6d2dc6c7f
Nov 03 14:19:59.172292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:59.172258    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4" event={"ID":"5b72d32b-da35-496d-a85e-327eac0da3f1","Type":"ContainerStarted","Data":"9d2636cc495f61c756e538e588f21ee840896a63740f187780bd86db75422496"}
Nov 03 14:19:59.173576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:59.173554    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v" event={"ID":"9c143c15-a7c2-4d1f-8e60-216aae706edf","Type":"ContainerStarted","Data":"9f9b3aa6bf2419df680643ec6356b5d290eb0556b37cb15712d43ba6d2dc6c7f"}
Nov 03 14:19:59.216971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:59.216927    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4" podStartSLOduration=0.950587099 podStartE2EDuration="2.216913968s" totalImagesPullingTime="1.266326869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:59.215884116 +0000 UTC m=+477.265915301" watchObservedRunningTime="2026-11-03 14:19:59.216913968 +0000 UTC m=+477.266945188"
Nov 03 14:19:59.505791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:59.505755    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:19:59.820935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:19:59.820836    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:00.180141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:00.180107    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v" event={"ID":"9c143c15-a7c2-4d1f-8e60-216aae706edf","Type":"ContainerStarted","Data":"edf9b0bcecaad2cbf998a313093afa5322d1f1ceecb1572f75f1ee6d89f2eda8"}
Nov 03 14:20:00.224713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:00.224653    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v" podStartSLOduration=2.028373676 podStartE2EDuration="2.224636902s" totalImagesPullingTime="196.263226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:00.223478124 +0000 UTC m=+478.273509307" watchObservedRunningTime="2026-11-03 14:20:00.224636902 +0000 UTC m=+478.274668086"
Nov 03 14:20:00.723573 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:00.723536    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:20:01.331155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:01.331115    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:02.352370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:02.352341    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:03.654675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:03.654641    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:20:05.100474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.100430    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"]
Nov 03 14:20:05.103477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.103445    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.128907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.128872    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"]
Nov 03 14:20:05.138070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.138032    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 14:20:05.177377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.177335    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x287m\" (UniqueName: \"kubernetes.io/projected/6212728d-6351-4e9b-bbfd-f9424c48412b-kube-api-access-x287m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.177580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.177483    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.177580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.177547    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6212728d-6351-4e9b-bbfd-f9424c48412b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.177580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.177574    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.177755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.177696    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.177828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.177806    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.278864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.278815    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6212728d-6351-4e9b-bbfd-f9424c48412b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.278864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.278865    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.278893    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.278923    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.279050    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x287m\" (UniqueName: \"kubernetes.io/projected/6212728d-6351-4e9b-bbfd-f9424c48412b-kube-api-access-x287m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.279104    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.279333    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.279424    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.279540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.279518    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.280069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.280043    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:05.281430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.281396    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6212728d-6351-4e9b-bbfd-f9424c48412b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.281614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.281594    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6212728d-6351-4e9b-bbfd-f9424c48412b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.306826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.306769    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x287m\" (UniqueName: \"kubernetes.io/projected/6212728d-6351-4e9b-bbfd-f9424c48412b-kube-api-access-x287m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts\" (UID: \"6212728d-6351-4e9b-bbfd-f9424c48412b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.418131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.418071    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:05.727971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:05.727939    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"]
Nov 03 14:20:05.730791 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:20:05.730750    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6212728d_6351_4e9b_bbfd_f9424c48412b.slice/crio-7afc63d43c99eef05e3da3daf13bdfb70bce85631ecf87c3117e0dfad1429bc2 WatchSource:0}: Error finding container 7afc63d43c99eef05e3da3daf13bdfb70bce85631ecf87c3117e0dfad1429bc2: Status 404 returned error can't find the container with id 7afc63d43c99eef05e3da3daf13bdfb70bce85631ecf87c3117e0dfad1429bc2
Nov 03 14:20:06.214719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:06.214681    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" event={"ID":"6212728d-6351-4e9b-bbfd-f9424c48412b","Type":"ContainerStarted","Data":"8c9ad72656b84bd4241a898f8df86356bcfa9214abd683f5736a5fcc9319fa39"}
Nov 03 14:20:06.214719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:06.214723    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" event={"ID":"6212728d-6351-4e9b-bbfd-f9424c48412b","Type":"ContainerStarted","Data":"7afc63d43c99eef05e3da3daf13bdfb70bce85631ecf87c3117e0dfad1429bc2"}
Nov 03 14:20:06.260256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:06.260117    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podStartSLOduration=1.062995389 podStartE2EDuration="1.260103101s" totalImagesPullingTime="197.107712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:06.259296568 +0000 UTC m=+484.309327752" watchObservedRunningTime="2026-11-03 14:20:06.260103101 +0000 UTC m=+484.310134284"
Nov 03 14:20:06.418523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:06.418486    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:07.506670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:07.506638    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:20:07.519001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:07.518978    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:20:08.235366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:08.235337    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-pbhb4"
Nov 03 14:20:08.723726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:08.723690    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:20:08.736220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:08.736194    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:20:09.242209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:09.242174    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tr28v"
Nov 03 14:20:10.941378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:10.941342    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:20:15.419294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:15.419256    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:15.432010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:15.431989    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:16.275696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:16.275667    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:20:19.715673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.715639    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-948677c5f-lc4kc"]
Nov 03 14:20:19.718194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.718172    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:19.740662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.740640    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-948677c5f-lc4kc"]
Nov 03 14:20:19.825412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.825378    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnm4\" (UniqueName: \"kubernetes.io/projected/1a4df69a-9878-4445-a3ab-94d60af00096-kube-api-access-ggnm4\") pod \"authorino-948677c5f-lc4kc\" (UID: \"1a4df69a-9878-4445-a3ab-94d60af00096\") " pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:19.825412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.825414    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4df69a-9878-4445-a3ab-94d60af00096-tls-cert\") pod \"authorino-948677c5f-lc4kc\" (UID: \"1a4df69a-9878-4445-a3ab-94d60af00096\") " pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:19.926761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.926715    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnm4\" (UniqueName: \"kubernetes.io/projected/1a4df69a-9878-4445-a3ab-94d60af00096-kube-api-access-ggnm4\") pod \"authorino-948677c5f-lc4kc\" (UID: \"1a4df69a-9878-4445-a3ab-94d60af00096\") " pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:19.926761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.926751    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4df69a-9878-4445-a3ab-94d60af00096-tls-cert\") pod \"authorino-948677c5f-lc4kc\" (UID: \"1a4df69a-9878-4445-a3ab-94d60af00096\") " pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:19.929163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.929142    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1a4df69a-9878-4445-a3ab-94d60af00096-tls-cert\") pod \"authorino-948677c5f-lc4kc\" (UID: \"1a4df69a-9878-4445-a3ab-94d60af00096\") " pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:19.948530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:19.948510    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnm4\" (UniqueName: \"kubernetes.io/projected/1a4df69a-9878-4445-a3ab-94d60af00096-kube-api-access-ggnm4\") pod \"authorino-948677c5f-lc4kc\" (UID: \"1a4df69a-9878-4445-a3ab-94d60af00096\") " pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:20.028821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:20.028753    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-948677c5f-lc4kc"
Nov 03 14:20:20.323782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:20.323758    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-948677c5f-lc4kc"]
Nov 03 14:20:20.325634 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:20:20.325599    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a4df69a_9878_4445_a3ab_94d60af00096.slice/crio-eb768781ae8fdbc22283f63aab553e780cf188525feea228feee231a80699eef WatchSource:0}: Error finding container eb768781ae8fdbc22283f63aab553e780cf188525feea228feee231a80699eef: Status 404 returned error can't find the container with id eb768781ae8fdbc22283f63aab553e780cf188525feea228feee231a80699eef
Nov 03 14:20:20.327095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:20.327077    6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:20:21.291087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.291051    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-948677c5f-lc4kc" event={"ID":"1a4df69a-9878-4445-a3ab-94d60af00096","Type":"ContainerStarted","Data":"b95ce408f6f52121a95615a2bc7868fe3d4fea6759588d6f46c062db6e3b9892"}
Nov 03 14:20:21.291087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.291092    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-948677c5f-lc4kc" event={"ID":"1a4df69a-9878-4445-a3ab-94d60af00096","Type":"ContainerStarted","Data":"eb768781ae8fdbc22283f63aab553e780cf188525feea228feee231a80699eef"}
Nov 03 14:20:21.332624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.332574    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-948677c5f-lc4kc" podStartSLOduration=1.505317145 podStartE2EDuration="2.332559489s" totalImagesPullingTime="827.242344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:21.331734533 +0000 UTC m=+499.381765716" watchObservedRunningTime="2026-11-03 14:20:21.332559489 +0000 UTC m=+499.382590672"
Nov 03 14:20:21.362717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.362687    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ffb77fcd4-8kmbf"]
Nov 03 14:20:21.362895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.362876    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf" podUID="5174748c-e439-4268-9ed8-92318a988c71" containerName="authorino" containerID="cri-o://033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470" gracePeriod=30
Nov 03 14:20:21.624395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.624364    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:20:21.744808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.744779    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw\" (UniqueName: \"kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw\") pod \"5174748c-e439-4268-9ed8-92318a988c71\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") "
Nov 03 14:20:21.745007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.744847    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert\" (UniqueName: \"kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert\") pod \"5174748c-e439-4268-9ed8-92318a988c71\" (UID: \"5174748c-e439-4268-9ed8-92318a988c71\") "
Nov 03 14:20:21.746987 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.746957    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw" pod "5174748c-e439-4268-9ed8-92318a988c71" (UID: "5174748c-e439-4268-9ed8-92318a988c71"). InnerVolumeSpecName "kube-api-access-x2rnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:20:21.755612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.755587    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert" pod "5174748c-e439-4268-9ed8-92318a988c71" (UID: "5174748c-e439-4268-9ed8-92318a988c71"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:20:21.845980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.845944    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rnw\" (UniqueName: \"kubernetes.io/projected/5174748c-e439-4268-9ed8-92318a988c71-kube-api-access-x2rnw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:20:21.845980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:21.845974    6481 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5174748c-e439-4268-9ed8-92318a988c71-tls-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:20:22.298889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.298845    6481 generic.go:356] "Generic (PLEG): container finished" podID="5174748c-e439-4268-9ed8-92318a988c71" containerID="033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470" exitCode=0
Nov 03 14:20:22.299376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.298923    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf"
Nov 03 14:20:22.299376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.298936    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf" event={"ID":"5174748c-e439-4268-9ed8-92318a988c71","Type":"ContainerDied","Data":"033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470"}
Nov 03 14:20:22.299376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.298982    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5ffb77fcd4-8kmbf" event={"ID":"5174748c-e439-4268-9ed8-92318a988c71","Type":"ContainerDied","Data":"f8280433e01061e92c01d3a6c0874d8514e602ad8d2a544aeee7ab1dd560d047"}
Nov 03 14:20:22.299376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.299003    6481 scope.go:122] "RemoveContainer" containerID="033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470"
Nov 03 14:20:22.310348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.310304    6481 scope.go:122] "RemoveContainer" containerID="033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470"
Nov 03 14:20:22.310623 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:20:22.310593    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470\": container with ID starting with 033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470 not found: ID does not exist" containerID="033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470"
Nov 03 14:20:22.310695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.310636    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470"} err="failed to get container status \"033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470\": rpc error: code = NotFound desc = could not find container \"033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470\": container with ID starting with 033cb637c5e32a6810c87f37e87085cba6c5193a6d65d2f0d79a8fe323edb470 not found: ID does not exist"
Nov 03 14:20:22.361540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.361462    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5ffb77fcd4-8kmbf"]
Nov 03 14:20:22.374090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.374064    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5ffb77fcd4-8kmbf"]
Nov 03 14:20:22.489456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.489424    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:22.735104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:22.735071    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:24.065659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:24.065623    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5174748c-e439-4268-9ed8-92318a988c71" path="/var/lib/kubelet/pods/5174748c-e439-4268-9ed8-92318a988c71/volumes"
Nov 03 14:20:24.089725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:24.089691    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:24.308558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:24.308518    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:27.984705 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:20:27.984670    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:20:33.464016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.463987    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"]
Nov 03 14:20:33.464424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.464410    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5174748c-e439-4268-9ed8-92318a988c71" containerName="authorino"
Nov 03 14:20:33.464505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.464425    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5174748c-e439-4268-9ed8-92318a988c71" containerName="authorino"
Nov 03 14:20:33.464505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.464488    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5174748c-e439-4268-9ed8-92318a988c71" containerName="authorino"
Nov 03 14:20:33.470216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.470194    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.501584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.501554    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"]
Nov 03 14:20:33.585424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585383    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585465    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585498    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585539    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585578    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xsl\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585608    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585658    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585722    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.585845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.585755    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686656    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686703    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686721    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686743    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686764    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xsl\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686784    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686810    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.686863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686850    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.687271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.686880    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.687271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.687154    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.687271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.687153    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.687465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.687295    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.687465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.687335    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.687656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.687633    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.689068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.689045    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.689160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.689143    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.707982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.707961    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.708387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.708363    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xsl\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-xrpdz\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:33.782924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:33.782860    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:34.100425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.100404    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"]
Nov 03 14:20:34.102647 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:20:34.102615    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b619dc_628e_4fec_b7dc_e8d936dd1a98.slice/crio-5ca788a911af56ab746e067f033eb519b99d3da327f7df7338582b3b31503e66 WatchSource:0}: Error finding container 5ca788a911af56ab746e067f033eb519b99d3da327f7df7338582b3b31503e66: Status 404 returned error can't find the container with id 5ca788a911af56ab746e067f033eb519b99d3da327f7df7338582b3b31503e66
Nov 03 14:20:34.104634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.104583    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:34.104714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.104668    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:34.104714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.104695    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:34.366395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.366317    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" event={"ID":"69b619dc-628e-4fec-b7dc-e8d936dd1a98","Type":"ContainerStarted","Data":"cdfc3cb509f189040d1328637d97b437aeab60587d3e808e6a79380ed0099b7e"}
Nov 03 14:20:34.366395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.366371    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" event={"ID":"69b619dc-628e-4fec-b7dc-e8d936dd1a98","Type":"ContainerStarted","Data":"5ca788a911af56ab746e067f033eb519b99d3da327f7df7338582b3b31503e66"}
Nov 03 14:20:34.412991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.412941    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podStartSLOduration=1.412926449 podStartE2EDuration="1.412926449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:34.411347131 +0000 UTC m=+512.461378317" watchObservedRunningTime="2026-11-03 14:20:34.412926449 +0000 UTC m=+512.462957610"
Nov 03 14:20:34.783978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:34.783944    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:35.784521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:35.784488    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:35.784910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:35.784547    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:36.783886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:36.783851    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:36.784062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:36.783907    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:37.784306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:37.784273    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:37.784693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:37.784326    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:38.784245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:38.784208    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:38.784428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:38.784286    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:39.784198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:39.784167    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:39.784397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:39.784221    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:40.783755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:40.783713    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:40.784299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:40.783765    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:41.783443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:41.783407    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Nov 03 14:20:41.783677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:41.783461    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Nov 03 14:20:43.253524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:43.253488    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:43.254040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:43.253929    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:43.254554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:43.254531    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:20:48.540130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.540102    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"]
Nov 03 14:20:48.542637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.542621    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.566101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.566073    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"]
Nov 03 14:20:48.647596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647562    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647604    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6clz\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647663    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647719    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647751    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647770    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647804    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647845    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.647955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.647924    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749277    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749315    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749354    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749386    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749423    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6clz\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749464    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749508    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749574    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749604    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.749833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749678    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.750003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.749978    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.750124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.750013    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.750124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.750076    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.750355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.750337    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.751510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.751492    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.751806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.751778    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.771443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.771415    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.771592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.771575    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6clz\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8jh2s\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:48.856479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:48.856401    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:49.162323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.162289    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"]
Nov 03 14:20:49.164831 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:20:49.164806    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76b3d2c_d2e0_4fcb_a06b_867851d1dd71.slice/crio-65355fa354c9c19de4df761b1049c737173267692d56785882a2523c1af6e28a WatchSource:0}: Error finding container 65355fa354c9c19de4df761b1049c737173267692d56785882a2523c1af6e28a: Status 404 returned error can't find the container with id 65355fa354c9c19de4df761b1049c737173267692d56785882a2523c1af6e28a
Nov 03 14:20:49.167023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.166994    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:49.167109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.167055    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:49.167144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.167109    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:49.437645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.437567    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" event={"ID":"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71","Type":"ContainerStarted","Data":"7b4f7a8c3348a04e71767e5a6e1042277f3ba8f8ab33c96815481c425ed36e94"}
Nov 03 14:20:49.437645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.437623    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" event={"ID":"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71","Type":"ContainerStarted","Data":"65355fa354c9c19de4df761b1049c737173267692d56785882a2523c1af6e28a"}
Nov 03 14:20:49.485453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.485398    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podStartSLOduration=1.485384044 podStartE2EDuration="1.485384044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:49.484295437 +0000 UTC m=+527.534326623" watchObservedRunningTime="2026-11-03 14:20:49.485384044 +0000 UTC m=+527.535415225"
Nov 03 14:20:49.856558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:49.856473    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:50.858515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:50.858484    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:50.858978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:50.858533    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:51.857298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:51.857259    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:51.857485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:51.857317    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:52.857473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:52.857436    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:52.857919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:52.857489    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:53.857147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:53.857101    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:53.857369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:53.857177    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:54.240398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.240355    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:54.581891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.581769    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"]
Nov 03 14:20:54.584843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.584821    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.607253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.606205    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\""
Nov 03 14:20:54.627259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.623455    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"]
Nov 03 14:20:54.730145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730104    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730360 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730310    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730408    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730449    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730512    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730541    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730569    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jjpd\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730599    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.730727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.730626    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.750097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.750062    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"]
Nov 03 14:20:54.753061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.753036    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.770843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.770821    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-156ae1-openshift-default-dockercfg-hh5m9\""
Nov 03 14:20:54.778633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.778602    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"]
Nov 03 14:20:54.834845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.834751    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.835626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.835602    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.835890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.835872    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.836098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.836071    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.836331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.836304    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.836526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.836507    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840347    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840442    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840480    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840508    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840535    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840567    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7hp5\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840623    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840653    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840681    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840710    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jjpd\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840738    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840762    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.840834    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.835503    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.841192    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.841463    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.841473    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.841625    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.843881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.843451    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.860262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.856770    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:54.860262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.856819    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:54.864601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.863984    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.869164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.869023    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jjpd\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.897164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.897137    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:54.951832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.951797    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.952582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.952559    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.953314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.953027    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.953724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.953681    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.953841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.953099    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.953988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.953975    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.954482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.954465    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.956134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.954805    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.956912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.956381    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.957097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.956692    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.957206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.957187    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.959386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.959366    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.959664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.959645    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7hp5\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.960124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.957610    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.963259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.957719    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.963259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.960495    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.977774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.977750    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:54.980537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:54.980501    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7hp5\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:55.068239 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.068192    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:55.280555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.280528    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"]
Nov 03 14:20:55.282162 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:20:55.282131    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5336c1d_06b1_40d0_b17a_b717e3ee753e.slice/crio-21b99e43b70e48da255ecd7950768d6f977abf66fae5f87dd3045364efcf74dc WatchSource:0}: Error finding container 21b99e43b70e48da255ecd7950768d6f977abf66fae5f87dd3045364efcf74dc: Status 404 returned error can't find the container with id 21b99e43b70e48da255ecd7950768d6f977abf66fae5f87dd3045364efcf74dc
Nov 03 14:20:55.284396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.284365    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:55.284498 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.284444    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:55.284555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.284518    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:55.377495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.377425    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"]
Nov 03 14:20:55.381970 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:20:55.381932    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585e5a88_4a71_4a33_9788_eebbe8356b8b.slice/crio-791eea1024ae5d8ce2c96cd621ffc5f354f03fc342c6f889ca0fcda5dda7d5dd WatchSource:0}: Error finding container 791eea1024ae5d8ce2c96cd621ffc5f354f03fc342c6f889ca0fcda5dda7d5dd: Status 404 returned error can't find the container with id 791eea1024ae5d8ce2c96cd621ffc5f354f03fc342c6f889ca0fcda5dda7d5dd
Nov 03 14:20:55.384754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.384716    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:55.384953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.384933    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:55.385053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.385041    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:20:55.468301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.468261    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" event={"ID":"585e5a88-4a71-4a33-9788-eebbe8356b8b","Type":"ContainerStarted","Data":"a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"}
Nov 03 14:20:55.468411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.468330    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" event={"ID":"585e5a88-4a71-4a33-9788-eebbe8356b8b","Type":"ContainerStarted","Data":"791eea1024ae5d8ce2c96cd621ffc5f354f03fc342c6f889ca0fcda5dda7d5dd"}
Nov 03 14:20:55.470154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.470118    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" event={"ID":"d5336c1d-06b1-40d0-b17a-b717e3ee753e","Type":"ContainerStarted","Data":"5a672c7976207f789092179921928671c21a257d813c995a0c46cabede8426ee"}
Nov 03 14:20:55.470359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.470193    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" event={"ID":"d5336c1d-06b1-40d0-b17a-b717e3ee753e","Type":"ContainerStarted","Data":"21b99e43b70e48da255ecd7950768d6f977abf66fae5f87dd3045364efcf74dc"}
Nov 03 14:20:55.562974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.562922    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podStartSLOduration=1.562907738 podStartE2EDuration="1.562907738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:55.56167298 +0000 UTC m=+533.611704171" watchObservedRunningTime="2026-11-03 14:20:55.562907738 +0000 UTC m=+533.612938920"
Nov 03 14:20:55.563165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.563051    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podStartSLOduration=1.56304501 podStartE2EDuration="1.56304501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:20:55.515190285 +0000 UTC m=+533.565221470" watchObservedRunningTime="2026-11-03 14:20:55.56304501 +0000 UTC m=+533.613076196"
Nov 03 14:20:55.758888 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.758853    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:55.857462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.857427    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:55.857596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.857492    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:55.897658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:55.897575    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:56.069309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.069219    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:56.075150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.075121    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:56.462129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.462099    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:56.475488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.475458    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:56.476506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.476490    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:20:56.857809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.857721    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Nov 03 14:20:56.857809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.857790    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Nov 03 14:20:56.872131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.872103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:56.903224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:56.903191    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:57.481654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:57.481609    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:57.482501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:57.482483    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:20:57.994983 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:20:57.994925    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:20:58.261792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.261674    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:58.492677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.492632    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:58.714146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.714034    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:20:58.857649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.857612    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:20:58.857843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.857671    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:20:58.923137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.923107    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:58.923530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.923511    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:20:58.924305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:20:58.924265    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:21:00.041840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.041809    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:00.210869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.210836    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"]
Nov 03 14:21:00.213449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.213431    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.229494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.229470    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgrs\" (UniqueName: \"kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs\") pod \"payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.229607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.229593    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.234849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.234826    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"]
Nov 03 14:21:00.252043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.252021    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-156ae1-dockercfg-6tv6h\""
Nov 03 14:21:00.252736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.252719    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-156ae1\""
Nov 03 14:21:00.260390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.260360    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:00.330555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.330483    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgrs\" (UniqueName: \"kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs\") pod \"payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.330711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.330565    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.331284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.331262    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.353466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.353438    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgrs\" (UniqueName: \"kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs\") pod \"payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.480516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.480471    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:00.526206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.526174    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:00.708424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.708394    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:00.820481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.820461    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"]
Nov 03 14:21:00.822921 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:00.822891    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a75e5cf_ac0f_491f_98fd_a8e3573e357e.slice/crio-c3a62f056577c899ae366a538d95500db6a476352c2da38db8d7e43023247d6c WatchSource:0}: Error finding container c3a62f056577c899ae366a538d95500db6a476352c2da38db8d7e43023247d6c: Status 404 returned error can't find the container with id c3a62f056577c899ae366a538d95500db6a476352c2da38db8d7e43023247d6c
Nov 03 14:21:00.929631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:00.929595    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:01.505758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.505728    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" event={"ID":"9a75e5cf-ac0f-491f-98fd-a8e3573e357e","Type":"ContainerStarted","Data":"c3a62f056577c899ae366a538d95500db6a476352c2da38db8d7e43023247d6c"}
Nov 03 14:21:01.871591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.871509    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:01.933212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.933180    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"]
Nov 03 14:21:01.935823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.935803    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:01.948790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.948768    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:01.948963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.948946    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-952tb\" (UniqueName: \"kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb\") pod \"payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:01.957076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.957018    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"]
Nov 03 14:21:01.967482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.967460    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-86c2e2-dockercfg-lstq7\""
Nov 03 14:21:01.967597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:01.967491    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-86c2e2\""
Nov 03 14:21:02.024942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.024912    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"]
Nov 03 14:21:02.029582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.029554    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.050556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.050517    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctv9q\" (UniqueName: \"kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q\") pod \"payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.050698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.050593    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.050763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.050693    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-952tb\" (UniqueName: \"kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb\") pod \"payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:02.050763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.050747    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:02.051224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.051199    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"]
Nov 03 14:21:02.065055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.065029    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-86c2e2\""
Nov 03 14:21:02.072083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.072061    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:02.076613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.076593    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-952tb\" (UniqueName: \"kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb\") pod \"payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:02.089502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.089478    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:02.155320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.154673    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctv9q\" (UniqueName: \"kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q\") pod \"payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.155320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.154717    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.155626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.155410    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.178798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.178775    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctv9q\" (UniqueName: \"kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q\") pod \"payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.265566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.265531    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-86c2e2-dockercfg-lstq7\""
Nov 03 14:21:02.271137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.271103    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:02.358462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.358430    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:02.582190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.582161    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"]
Nov 03 14:21:02.584809 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:02.584776    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc900825a_60e4_4b3b_ab2d_a98b10460b97.slice/crio-950efcd5721b8879814b286472c9b3274ae16d2dd7a0350d185d4883f43de986 WatchSource:0}: Error finding container 950efcd5721b8879814b286472c9b3274ae16d2dd7a0350d185d4883f43de986: Status 404 returned error can't find the container with id 950efcd5721b8879814b286472c9b3274ae16d2dd7a0350d185d4883f43de986
Nov 03 14:21:02.669730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.669701    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"]
Nov 03 14:21:02.671471 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:02.671446    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd80f7f_50a8_4b3b_8fa2_595dfec95e25.slice/crio-3ce8cb8a1691904e3f01b3b0a70ff7643c3bae3ec2361a12ce5ef11e8189b77e WatchSource:0}: Error finding container 3ce8cb8a1691904e3f01b3b0a70ff7643c3bae3ec2361a12ce5ef11e8189b77e: Status 404 returned error can't find the container with id 3ce8cb8a1691904e3f01b3b0a70ff7643c3bae3ec2361a12ce5ef11e8189b77e
Nov 03 14:21:02.710125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.710063    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:02.935470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.935393    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:02.992979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.992949    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"]
Nov 03 14:21:02.995591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:02.995567    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.017904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.017642    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4429aa\""
Nov 03 14:21:03.019070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.019025    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"]
Nov 03 14:21:03.019070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.019054    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4429aa-dockercfg-6qg29\""
Nov 03 14:21:03.094482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.092946    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"]
Nov 03 14:21:03.094992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.094966    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfdk\" (UniqueName: \"kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk\") pod \"payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.095433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.095390    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.097828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.097808    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.108412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.108195    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"]
Nov 03 14:21:03.175324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.175300    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:03.196418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.196368    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfdk\" (UniqueName: \"kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk\") pod \"payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.196533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.196457    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.196972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.196951    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7tq4\" (UniqueName: \"kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4\") pod \"payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.197210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.197188    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.197776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.197754    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.219872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.219844    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfdk\" (UniqueName: \"kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk\") pod \"payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.297889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.297864    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7tq4\" (UniqueName: \"kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4\") pod \"payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.298039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.297959    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.298611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.298593    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.312726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.312702    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:03.321341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.321322    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7tq4\" (UniqueName: \"kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4\") pod \"payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.442490 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.442462    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:03.518203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.518172    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" event={"ID":"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25","Type":"ContainerStarted","Data":"dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2"}
Nov 03 14:21:03.518351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.518254    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:03.518351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.518277    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" event={"ID":"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25","Type":"ContainerStarted","Data":"3ce8cb8a1691904e3f01b3b0a70ff7643c3bae3ec2361a12ce5ef11e8189b77e"}
Nov 03 14:21:03.519937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.519910    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" event={"ID":"c900825a-60e4-4b3b-ab2d-a98b10460b97","Type":"ContainerStarted","Data":"2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8"}
Nov 03 14:21:03.520061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.519957    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" event={"ID":"c900825a-60e4-4b3b-ab2d-a98b10460b97","Type":"ContainerStarted","Data":"950efcd5721b8879814b286472c9b3274ae16d2dd7a0350d185d4883f43de986"}
Nov 03 14:21:03.520131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.520089    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:03.521543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.521522    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" event={"ID":"9a75e5cf-ac0f-491f-98fd-a8e3573e357e","Type":"ContainerStarted","Data":"79438afb8ff6dfa0b7253a0b079bf4956399966dc8e960159908335b5fda6896"}
Nov 03 14:21:03.521981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.521964    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:03.567900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.567851    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" podStartSLOduration=2.236522358 podStartE2EDuration="2.567837537s" totalImagesPullingTime="331.315179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:03.566339372 +0000 UTC m=+541.616370557" watchObservedRunningTime="2026-11-03 14:21:03.567837537 +0000 UTC m=+541.617868759"
Nov 03 14:21:03.616446 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:03.616412    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c1f6f2_a8b2_42d7_adbe_0a3e2e52dbf0.slice/crio-9aab589d44567f9d0e29ef83466b93aba6713368d2b5724bf63c6fa8a12b24a0 WatchSource:0}: Error finding container 9aab589d44567f9d0e29ef83466b93aba6713368d2b5724bf63c6fa8a12b24a0: Status 404 returned error can't find the container with id 9aab589d44567f9d0e29ef83466b93aba6713368d2b5724bf63c6fa8a12b24a0
Nov 03 14:21:03.616884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.616476    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" podStartSLOduration=1.435915362 podStartE2EDuration="3.616462147s" totalImagesPullingTime="2.180546785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:03.612934324 +0000 UTC m=+541.662965507" watchObservedRunningTime="2026-11-03 14:21:03.616462147 +0000 UTC m=+541.666493330"
Nov 03 14:21:03.617061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.617039    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"]
Nov 03 14:21:03.656266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.656199    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" podStartSLOduration=2.198513779 podStartE2EDuration="2.656188095s" totalImagesPullingTime="457.674316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:03.655770996 +0000 UTC m=+541.705802179" watchObservedRunningTime="2026-11-03 14:21:03.656188095 +0000 UTC m=+541.706219279"
Nov 03 14:21:03.741439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:03.741416    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"]
Nov 03 14:21:03.743419 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:03.743395    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f55d01a_43dc_40d3_9729_ec4b9e6b57b6.slice/crio-bd1cc8a4b8de79d3f6835545a75a220c2b1577ab684426f812d1a24870aa29c9 WatchSource:0}: Error finding container bd1cc8a4b8de79d3f6835545a75a220c2b1577ab684426f812d1a24870aa29c9: Status 404 returned error can't find the container with id bd1cc8a4b8de79d3f6835545a75a220c2b1577ab684426f812d1a24870aa29c9
Nov 03 14:21:04.379298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.369216    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:04.533623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.533583    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" event={"ID":"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6","Type":"ContainerStarted","Data":"ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280"}
Nov 03 14:21:04.533900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.533874    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" event={"ID":"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6","Type":"ContainerStarted","Data":"bd1cc8a4b8de79d3f6835545a75a220c2b1577ab684426f812d1a24870aa29c9"}
Nov 03 14:21:04.534034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.534020    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:04.536427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.535776    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" event={"ID":"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0","Type":"ContainerStarted","Data":"97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f"}
Nov 03 14:21:04.536427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.535856    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:04.536427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.535881    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" event={"ID":"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0","Type":"ContainerStarted","Data":"9aab589d44567f9d0e29ef83466b93aba6713368d2b5724bf63c6fa8a12b24a0"}
Nov 03 14:21:04.576438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.576383    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" podStartSLOduration=1.576368856 podStartE2EDuration="1.576368856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:04.575542729 +0000 UTC m=+542.625573914" watchObservedRunningTime="2026-11-03 14:21:04.576368856 +0000 UTC m=+542.626400038"
Nov 03 14:21:04.622846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.622794    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" podStartSLOduration=2.622780597 podStartE2EDuration="2.622780597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:04.621348138 +0000 UTC m=+542.671379323" watchObservedRunningTime="2026-11-03 14:21:04.622780597 +0000 UTC m=+542.672811774"
Nov 03 14:21:04.880277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:04.880243    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:05.097445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:05.097412    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:06.919108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:06.919073    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:07.720333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:07.720290    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:08.926911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:08.926882    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:10.975540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:10.975510    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:12.524393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:12.524354    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:21:12.525048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:12.525017    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:21:13.167752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:13.167653    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:13.391897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:13.391869    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:13.688379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:13.688346    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:13.920580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:13.920549    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:13.979196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:13.979115    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:14.395058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:14.394983    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:14.539220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:14.539186    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:21:14.539377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:14.539286    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:21:15.524902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.524790    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:15.545883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.545857    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:21:15.546027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.546016    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:21:15.633996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.633961    6481 generic.go:356] "Generic (PLEG): container finished" podID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerID="a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12" exitCode=0
Nov 03 14:21:15.634160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.634039    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" event={"ID":"585e5a88-4a71-4a33-9788-eebbe8356b8b","Type":"ContainerDied","Data":"a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"}
Nov 03 14:21:15.634581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.634558    6481 scope.go:122] "RemoveContainer" containerID="a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"
Nov 03 14:21:15.636008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.635981    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:15.636131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.636058    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:15.636131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.636108    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:15.930697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.930664    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"]
Nov 03 14:21:15.934875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.934853    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:15.944999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.944977    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"]
Nov 03 14:21:15.952520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.952496    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"default-dockercfg-wqn4r\""
Nov 03 14:21:15.952637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.952534    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"simulator-w3-156ae1-kserve-self-signed-certs\""
Nov 03 14:21:15.952814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.952798    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"openshift-service-ca.crt\""
Nov 03 14:21:15.952915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:15.952898    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"kube-root-ca.crt\""
Nov 03 14:21:16.065244 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.065198    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.065440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.065339    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.065514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.065455    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.065559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.065510    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctz5g\" (UniqueName: \"kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.065640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.065621    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.065676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.065649    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.084306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.084273    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.166776    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctz5g\" (UniqueName: \"kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.166844    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.166860    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.166888    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.166935    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.166984    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.167404    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.167947    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.168817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.168325    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.171679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.171653    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.173834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.173793    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.192539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.192467    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctz5g\" (UniqueName: \"kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g\") pod \"simulator-w3-156ae1-kserve-95c9cb797-nd96l\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") " pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.253462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.253420    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:16.552686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.552607    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:21:16.608252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.600211    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"]
Nov 03 14:21:16.666523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.666491    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" event={"ID":"52ad424d-51fc-400c-bb0e-b7c6defea720","Type":"ContainerStarted","Data":"5be47fb557293a90343e60ac87067b03b0d8041e83e94a77a869e0f2f039c17e"}
Nov 03 14:21:16.668841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:16.668810    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" event={"ID":"585e5a88-4a71-4a33-9788-eebbe8356b8b","Type":"ContainerStarted","Data":"e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be"}
Nov 03 14:21:17.068748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.068665    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:21:17.069006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.068935    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 14:21:17.069006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.068984    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 14:21:17.594955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.594916    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"]
Nov 03 14:21:17.598920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.598901    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.628055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.628026    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"]
Nov 03 14:21:17.628517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.628499    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:21:17.628817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.628794    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:21:17.629003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.628974    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:21:17.629250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.629216    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:21:17.677447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.677410    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" event={"ID":"52ad424d-51fc-400c-bb0e-b7c6defea720","Type":"ContainerStarted","Data":"b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a"}
Nov 03 14:21:17.684549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.684517    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8hj\" (UniqueName: \"kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.684783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.684737    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.684890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.684868    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.741286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.741207    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:17.786169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.786136    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8hj\" (UniqueName: \"kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.786378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.786327    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.786472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.786424    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.787343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.787318    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.788896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.788864    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.831801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.831765    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8hj\" (UniqueName: \"kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj\") pod \"maas-api-revoke-sub-691d57b2-8z2qd\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:17.938264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:17.937973    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:18.069429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.069391    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 14:21:18.069684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.069635    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 14:21:18.254902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.254829    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:18.256081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.256051    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:18.324418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.297058    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podStartSLOduration=3.297036739 podStartE2EDuration="3.297036739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:17.739864867 +0000 UTC m=+555.789896052" watchObservedRunningTime="2026-11-03 14:21:18.297036739 +0000 UTC m=+556.347067926"
Nov 03 14:21:18.324418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.299542    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"]
Nov 03 14:21:18.688002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.687957    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd" event={"ID":"c062733e-0431-47ee-a345-d74e51932672","Type":"ContainerStarted","Data":"79920be692bc039615fda974ab502b0301e2dbb524fe98e0f62c39ab777e77fe"}
Nov 03 14:21:18.749328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.749264    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:18.973894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:18.973787    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:19.068780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:19.068736    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 14:21:19.068983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:19.068803    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 14:21:19.196160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:19.196122    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:20.027253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:20.010914    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:20.090929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:20.079201    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 14:21:20.090929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:20.079278    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 14:21:20.839065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:20.838791    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:21.071781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.071737    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 14:21:21.072522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.072472    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 14:21:21.185247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.185196    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"]
Nov 03 14:21:21.331443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.329739    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"]
Nov 03 14:21:21.331443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.329891    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.349058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.348215    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-0a0ec6-openshift-default-dockercfg-smnvs\""
Nov 03 14:21:21.509437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.496437    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:21.512779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.512743    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.541431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.541391    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.541691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.541669    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.541864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.541844    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5h4\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.545350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.545321    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.568372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.568332    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.569497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.569465    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.569701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.569683    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.569843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.569826    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.670826    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.670897    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.670947    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5h4\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671000    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671036    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671115    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671163    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671196    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671254    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.671738    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.672013    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.674266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.672664    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.692284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.683824    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.693014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.692965    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.705304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.702795    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5h4\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.705304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.703292    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.709462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.709440    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.727258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.724161    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy\") pod \"e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") " pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:21.729349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.729193    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:21.848476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:21.848435    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:22.075303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.074614    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Nov 03 14:21:22.075303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.074687    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Nov 03 14:21:22.647264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.637908    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:22.690707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.690680    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"]
Nov 03 14:21:22.694516 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:22.694483    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913318d5_bb9b_407e_8ba0_77f611c482a6.slice/crio-3a5e20a07f42ce930f3562d1544780732b954c0eba570e1420614e2325b34ea2 WatchSource:0}: Error finding container 3a5e20a07f42ce930f3562d1544780732b954c0eba570e1420614e2325b34ea2: Status 404 returned error can't find the container with id 3a5e20a07f42ce930f3562d1544780732b954c0eba570e1420614e2325b34ea2
Nov 03 14:21:22.712491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.710879    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:22.712491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.710974    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:22.712491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.711022    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:22.816450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:22.816347    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" event={"ID":"913318d5-bb9b-407e-8ba0-77f611c482a6","Type":"ContainerStarted","Data":"3a5e20a07f42ce930f3562d1544780732b954c0eba570e1420614e2325b34ea2"}
Nov 03 14:21:23.373149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.373118    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:21:23.373630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.373611    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:21:23.375375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.375355    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:21:23.827489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.827447    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" event={"ID":"913318d5-bb9b-407e-8ba0-77f611c482a6","Type":"ContainerStarted","Data":"9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"}
Nov 03 14:21:23.849357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.849324    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:23.855462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.855437    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:23.876258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:23.874041    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podStartSLOduration=2.8740257959999997 podStartE2EDuration="2.874025796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:23.873579982 +0000 UTC m=+561.923611164" watchObservedRunningTime="2026-11-03 14:21:23.874025796 +0000 UTC m=+561.924056979"
Nov 03 14:21:24.247547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:24.247517    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:24.621217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:24.621181    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:24.834473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:24.834427    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd" event={"ID":"c062733e-0431-47ee-a345-d74e51932672","Type":"ContainerStarted","Data":"7f530e08ee92bcc02837339624e702545c3ea678cd42baf0b905d0b770dc7f61"}
Nov 03 14:21:24.835791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:24.835007    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:24.836025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:24.836003    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:24.876900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:24.876851    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd" podStartSLOduration=2.048351353 podStartE2EDuration="7.876837764s" totalImagesPullingTime="5.828486411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:24.874884413 +0000 UTC m=+562.924915598" watchObservedRunningTime="2026-11-03 14:21:24.876837764 +0000 UTC m=+562.926868947"
Nov 03 14:21:25.841204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.841173    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerID="5a672c7976207f789092179921928671c21a257d813c995a0c46cabede8426ee" exitCode=0
Nov 03 14:21:25.841664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.841256    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" event={"ID":"d5336c1d-06b1-40d0-b17a-b717e3ee753e","Type":"ContainerDied","Data":"5a672c7976207f789092179921928671c21a257d813c995a0c46cabede8426ee"}
Nov 03 14:21:25.841734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.841666    6481 scope.go:122] "RemoveContainer" containerID="5a672c7976207f789092179921928671c21a257d813c995a0c46cabede8426ee"
Nov 03 14:21:25.842924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.842901    6481 generic.go:356] "Generic (PLEG): container finished" podID="c062733e-0431-47ee-a345-d74e51932672" containerID="7f530e08ee92bcc02837339624e702545c3ea678cd42baf0b905d0b770dc7f61" exitCode=0
Nov 03 14:21:25.843014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.842985    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd" event={"ID":"c062733e-0431-47ee-a345-d74e51932672","Type":"ContainerDied","Data":"7f530e08ee92bcc02837339624e702545c3ea678cd42baf0b905d0b770dc7f61"}
Nov 03 14:21:25.843054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.842990    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:25.843096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.843081    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:25.843148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:25.843136    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:26.254335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.254302    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:26.268479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.268452    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:26.453836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.453793    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:26.697288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.697260    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:26.850037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.850001    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" event={"ID":"d5336c1d-06b1-40d0-b17a-b717e3ee753e","Type":"ContainerStarted","Data":"1394ac8605926b72897342f7b5903d4bcaa847ba54626e50bafb30e0f156ece9"}
Nov 03 14:21:26.863066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.863031    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:21:26.897320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.897286    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:21:26.898487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.898458    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 14:21:26.898601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.898530    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 14:21:26.937295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:26.937258    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:27.039288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.039260    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:27.058469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.058442    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca\") pod \"c062733e-0431-47ee-a345-d74e51932672\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") "
Nov 03 14:21:27.058596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.058491    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca\") pod \"c062733e-0431-47ee-a345-d74e51932672\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") "
Nov 03 14:21:27.058596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.058519    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj\" (UniqueName: \"kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj\") pod \"c062733e-0431-47ee-a345-d74e51932672\" (UID: \"c062733e-0431-47ee-a345-d74e51932672\") "
Nov 03 14:21:27.058771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.058747    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca" pod "c062733e-0431-47ee-a345-d74e51932672" (UID: "c062733e-0431-47ee-a345-d74e51932672"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:21:27.060614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.060592    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj" pod "c062733e-0431-47ee-a345-d74e51932672" (UID: "c062733e-0431-47ee-a345-d74e51932672"). InnerVolumeSpecName "kube-api-access-4l8hj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:21:27.060697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.060613    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca" pod "c062733e-0431-47ee-a345-d74e51932672" (UID: "c062733e-0431-47ee-a345-d74e51932672"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:21:27.159970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.159934    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c062733e-0431-47ee-a345-d74e51932672-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:21:27.159970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.159961    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8hj\" (UniqueName: \"kubernetes.io/projected/c062733e-0431-47ee-a345-d74e51932672-kube-api-access-4l8hj\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:21:27.159970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.159972    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c062733e-0431-47ee-a345-d74e51932672-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:21:27.482482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.482451    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:21:27.861551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.861378    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd" event={"ID":"c062733e-0431-47ee-a345-d74e51932672","Type":"ContainerDied","Data":"79920be692bc039615fda974ab502b0301e2dbb524fe98e0f62c39ab777e77fe"}
Nov 03 14:21:27.862281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.862256    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79920be692bc039615fda974ab502b0301e2dbb524fe98e0f62c39ab777e77fe"
Nov 03 14:21:27.862442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.862272    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"
Nov 03 14:21:27.897453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.897429    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 14:21:27.897556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:27.897474    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 14:21:27.986245 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:21:27.986190    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:21:28.263809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.263780    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:28.455095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.455063    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"]
Nov 03 14:21:28.458421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.458379    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c062733e-0431-47ee-a345-d74e51932672" containerName="revoke-keys"
Nov 03 14:21:28.458421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.458421    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c062733e-0431-47ee-a345-d74e51932672" containerName="revoke-keys"
Nov 03 14:21:28.458642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.458608    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c062733e-0431-47ee-a345-d74e51932672" containerName="revoke-keys"
Nov 03 14:21:28.466714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.466430    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.475189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.475157    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"]
Nov 03 14:21:28.483666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.482366    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:28.486913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.486890    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0a0ec6\""
Nov 03 14:21:28.487056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.486992    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0a0ec6-dockercfg-brnqg\""
Nov 03 14:21:28.580098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.580013    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvv4\" (UniqueName: \"kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4\") pod \"payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.580098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.580050    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.681208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.681175    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvv4\" (UniqueName: \"kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4\") pod \"payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.681208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.681214    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.681992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.681974    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.731814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.729846    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvv4\" (UniqueName: \"kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4\") pod \"payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.789436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.789400    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:28.838286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.838157    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:28.898945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.898910    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 14:21:28.899294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:28.898971    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 14:21:29.125977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.122878    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"]
Nov 03 14:21:29.132417 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:29.132383    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844db57e_253a_4573_978d_48222f1e2cc9.slice/crio-80a9e4422ad2878753c24e341c763ba74daa9c2bcb385cb6db757e71afdd9108 WatchSource:0}: Error finding container 80a9e4422ad2878753c24e341c763ba74daa9c2bcb385cb6db757e71afdd9108: Status 404 returned error can't find the container with id 80a9e4422ad2878753c24e341c763ba74daa9c2bcb385cb6db757e71afdd9108
Nov 03 14:21:29.875096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.875064    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk" event={"ID":"844db57e-253a-4573-978d-48222f1e2cc9","Type":"ContainerStarted","Data":"427646f4b5dd4d6604f7471d3f24de32cf205b8611f5a9a0de49e0618b061f72"}
Nov 03 14:21:29.875290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.875113    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk" event={"ID":"844db57e-253a-4573-978d-48222f1e2cc9","Type":"ContainerStarted","Data":"80a9e4422ad2878753c24e341c763ba74daa9c2bcb385cb6db757e71afdd9108"}
Nov 03 14:21:29.875290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.875176    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:29.898397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.898370    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 14:21:29.898517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.898420    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 14:21:29.934099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:29.934046    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk" podStartSLOduration=1.934031193 podStartE2EDuration="1.934031193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:29.919753015 +0000 UTC m=+567.969784200" watchObservedRunningTime="2026-11-03 14:21:29.934031193 +0000 UTC m=+567.984062375"
Nov 03 14:21:30.811946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:30.811900    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:30.898022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:30.897991    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Nov 03 14:21:30.898180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:30.898038    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Nov 03 14:21:32.565286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.565251    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:21:32.567272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.567239    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:21:32.727807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.727768    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:32.951838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.951809    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"]
Nov 03 14:21:32.955457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.955432    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:32.976462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.976416    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-156ae1\"/\"premium-w3-156ae1-kserve-self-signed-certs\""
Nov 03 14:21:32.978322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:32.978297    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"]
Nov 03 14:21:33.057000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.056960    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.057000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.057000    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.057198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.057104    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.057198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.057161    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddbtl\" (UniqueName: \"kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.057284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.057239    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.057318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.057296    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.134272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.134212    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:33.158784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.158744    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.160186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160161    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.160493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160475    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.160648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160632    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddbtl\" (UniqueName: \"kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.160864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160849    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.161042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160998    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.161485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160851    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.161485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.161052    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.161793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.160004    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.164639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.164595    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.165089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.165068    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.182975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.182949    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddbtl\" (UniqueName: \"kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl\") pod \"premium-w3-156ae1-kserve-7688f598b4-8s4bb\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") " pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.268529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.268454    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:33.586189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.586164    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"]
Nov 03 14:21:33.587791 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:33.587765    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61f10c56_c20d_4e6f_b0c2_d43c6487eabd.slice/crio-580a750985443aecbd50bc483a671157b2f0eabf380d3c2b5aa90819a85452fb WatchSource:0}: Error finding container 580a750985443aecbd50bc483a671157b2f0eabf380d3c2b5aa90819a85452fb: Status 404 returned error can't find the container with id 580a750985443aecbd50bc483a671157b2f0eabf380d3c2b5aa90819a85452fb
Nov 03 14:21:33.644176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.644150    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:33.672794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.672764    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"]
Nov 03 14:21:33.675984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.675963    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.697843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.697816    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"]
Nov 03 14:21:33.768391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768357    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768393    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768519    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768548    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768572    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768686    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768719    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768737    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvgdv\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.768889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.768870    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.869904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.869821    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.869904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.869854    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.869904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.869877    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.869925    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.869960    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.869981    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvgdv\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870052    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870100    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870129    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870317    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870501    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870537    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.870660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.870635    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.871083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.871062    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.872280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.872259    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.872534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.872518    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.890999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.890974    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.891370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.891355    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvgdv\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-6d2jm\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:33.898279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.898257    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" event={"ID":"61f10c56-c20d-4e6f-b0c2-d43c6487eabd","Type":"ContainerStarted","Data":"6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806"}
Nov 03 14:21:33.898372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.898303    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" event={"ID":"61f10c56-c20d-4e6f-b0c2-d43c6487eabd","Type":"ContainerStarted","Data":"580a750985443aecbd50bc483a671157b2f0eabf380d3c2b5aa90819a85452fb"}
Nov 03 14:21:33.940944 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.940897    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podStartSLOduration=1.940882608 podStartE2EDuration="1.940882608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:33.940537744 +0000 UTC m=+571.990568927" watchObservedRunningTime="2026-11-03 14:21:33.940882608 +0000 UTC m=+571.990913824"
Nov 03 14:21:33.988989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:33.988965    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:34.269369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.269341    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:34.298504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.298482    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"]
Nov 03 14:21:34.299932 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:34.299903    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8a4a1c_82ad_4482_b4a1_7c64c6bea25a.slice/crio-fd0d36a8a11fc3f988f906b4ceb816a81fed6f3f698d65820c41fc395b5d36d6 WatchSource:0}: Error finding container fd0d36a8a11fc3f988f906b4ceb816a81fed6f3f698d65820c41fc395b5d36d6: Status 404 returned error can't find the container with id fd0d36a8a11fc3f988f906b4ceb816a81fed6f3f698d65820c41fc395b5d36d6
Nov 03 14:21:34.302175 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.302140    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:34.302297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.302204    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:34.302297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.302246    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:34.790691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.790656    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:34.927787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.927734    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" event={"ID":"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a","Type":"ContainerStarted","Data":"26e2575d19d1f76c33ea99a064d68fc122f1b7b007a158ce94cfccf063841660"}
Nov 03 14:21:34.928002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.927834    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" event={"ID":"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a","Type":"ContainerStarted","Data":"fd0d36a8a11fc3f988f906b4ceb816a81fed6f3f698d65820c41fc395b5d36d6"}
Nov 03 14:21:34.988356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.988291    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podStartSLOduration=1.988275334 podStartE2EDuration="1.988275334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:34.988029155 +0000 UTC m=+573.038060339" watchObservedRunningTime="2026-11-03 14:21:34.988275334 +0000 UTC m=+573.038306520"
Nov 03 14:21:34.990082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:34.990065    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:35.025852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:35.025813    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:35.991636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:35.991599    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:35.992013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:35.991653    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:36.925698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:36.925657    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:36.989765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:36.989729    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:36.989942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:36.989781    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:37.989676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:37.989645    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:37.990075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:37.989699    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:38.989947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:38.989872    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:38.990668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:38.990638    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:39.056194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:39.056105    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:39.990288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:39.989683    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:39.990288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:39.989746    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:40.883924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:40.883892    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:21:40.989539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:40.989506    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:40.989747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:40.989578    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:41.116254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:41.115007    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:41.990358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:41.990319    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Nov 03 14:21:41.990529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:41.990391    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Nov 03 14:21:43.091616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:43.091583    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:43.269289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:43.269258    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:43.282737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:43.282706    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:44.031457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.031410    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:21:44.031668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.031485    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:21:44.037853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.037816    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:21:44.094135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.094099    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:44.280221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.280190    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:44.345918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.345831    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:44.346549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.346521    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:44.347308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.347290    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:21:44.556303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.556273    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:44.827728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:44.827693    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:45.117000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:45.116879    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:47.898991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:47.898953    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:48.293735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.293630    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:48.519255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.517120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:48.815873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.815834    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"]
Nov 03 14:21:48.820386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.820358    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.843827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.843796    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"]
Nov 03 14:21:48.859273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.854472    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"default-dockercfg-xchvf\""
Nov 03 14:21:48.859273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.854554    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"premium-w0-86c2e2-kserve-self-signed-certs\""
Nov 03 14:21:48.859273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.855133    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"openshift-service-ca.crt\""
Nov 03 14:21:48.859273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.855148    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"kube-root-ca.crt\""
Nov 03 14:21:48.936962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.936925    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.937400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.937050    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.937400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.937093    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf5s5\" (UniqueName: \"kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.937400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.937224    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.937400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.937280    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.937400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.937317    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:48.987823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:48.987793    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.038825    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.042898    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.042969    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf5s5\" (UniqueName: \"kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.043115    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.043160    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.043215    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.040083    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.047945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.047595    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.048025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.047957    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.051692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.051658    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.054925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.054898    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.068864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.068805    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf5s5\" (UniqueName: \"kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5\") pod \"premium-w0-86c2e2-kserve-5995d88d58-vg9dg\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") " pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.136669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.136631    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:49.437863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:49.437835    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"]
Nov 03 14:21:49.439589 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:21:49.439558    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb63f6ab_d548_4c8f_8ff7_0615ca78b4a2.slice/crio-72e5c62264765f0bde283c23ec6410f4c9a14eab273eb804b026386612a1ad87 WatchSource:0}: Error finding container 72e5c62264765f0bde283c23ec6410f4c9a14eab273eb804b026386612a1ad87: Status 404 returned error can't find the container with id 72e5c62264765f0bde283c23ec6410f4c9a14eab273eb804b026386612a1ad87
Nov 03 14:21:50.062707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.062676    6481 generic.go:356] "Generic (PLEG): container finished" podID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerID="9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6" exitCode=0
Nov 03 14:21:50.066395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.066366    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" event={"ID":"913318d5-bb9b-407e-8ba0-77f611c482a6","Type":"ContainerDied","Data":"9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"}
Nov 03 14:21:50.066510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.066421    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" event={"ID":"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2","Type":"ContainerStarted","Data":"7bef5cd9a2790019a9ea48be993d280757119001888716148f6f43c0f33b103d"}
Nov 03 14:21:50.066510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.066440    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" event={"ID":"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2","Type":"ContainerStarted","Data":"72e5c62264765f0bde283c23ec6410f4c9a14eab273eb804b026386612a1ad87"}
Nov 03 14:21:50.066719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.066706    6481 scope.go:122] "RemoveContainer" containerID="9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"
Nov 03 14:21:50.067847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.067819    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:50.067933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.067885    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:50.067933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.067915    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:21:50.068614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.068570    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:50.110432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.110388    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podStartSLOduration=2.11037441 podStartE2EDuration="2.11037441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:50.108901241 +0000 UTC m=+588.158932425" watchObservedRunningTime="2026-11-03 14:21:50.11037441 +0000 UTC m=+588.160405593"
Nov 03 14:21:50.137491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.137469    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:50.302395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.302323    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:50.528493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:50.528459    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:51.073015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:51.072981    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" event={"ID":"913318d5-bb9b-407e-8ba0-77f611c482a6","Type":"ContainerStarted","Data":"de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6"}
Nov 03 14:21:51.599843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:51.599813    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:51.856005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:51.855491    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:51.856005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:51.855619    6481 patch_prober.go:29] interesting pod/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 14:21:51.856005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:51.855667    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 14:21:52.421250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:52.414289    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:52.849554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:52.849473    6481 patch_prober.go:29] interesting pod/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 14:21:52.849554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:52.849540    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 14:21:53.849826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:53.849785    6481 patch_prober.go:29] interesting pod/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 14:21:53.850214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:53.849847    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 14:21:54.326584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:54.326550    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:54.835196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:54.835156    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:54.850035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:54.850002    6481 patch_prober.go:29] interesting pod/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 14:21:54.850432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:54.850055    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 14:21:55.849704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:55.849672    6481 patch_prober.go:29] interesting pod/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Nov 03 14:21:55.849901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:55.849724    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Nov 03 14:21:56.307973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:56.307938    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:56.659683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:56.659640    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:56.853503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:56.853476    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:56.854630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:56.854609    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:21:57.467949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:57.467907    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:21:57.989493 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:21:57.989446    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:21:58.296322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:58.291412    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:58.536246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:58.536187    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:58.791392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:58.791359    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:21:59.137476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:59.137439    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:21:59.149594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:21:59.149574    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:22:00.147523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:00.147494    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:22:00.427173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:00.427141    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:02.421724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:02.421683    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:03.630583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:03.630542    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:04.418666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:04.418631    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:06.427890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:06.427855    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:06.793041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:06.792966    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:07.012864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:07.012827    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:08.402324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:08.402293    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:10.407143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:10.407108    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:11.000261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:11.000217    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:22:12.434335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:12.434299    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:13.288400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:13.288370    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:13.734903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:13.734868    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:14.029733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.029662    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"]
Nov 03 14:22:14.035105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.035084    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.072571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.072542    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2d0da2-openshift-default-dockercfg-288bc\""
Nov 03 14:22:14.076990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.076965    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"]
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.176782    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7sgz\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.176843    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.176881    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.176918    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.176951    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.176998    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.177042    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.177078    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.177458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.177126    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.204549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.204489    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:14.278563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278529    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.278736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278581    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.278736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278711    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7sgz\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.278842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278749    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.278842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278773    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.278842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278805    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.278842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278837    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278954    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.278996    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.279020    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.279129    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.279343    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.279408    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.279673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.279651    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.281272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.281210    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.281379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.281361    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.300393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.300366    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.300720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.300703    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7sgz\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.349042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.349020    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:14.430315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.430281    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:14.646843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.646772    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:14.688363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.688342    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"]
Nov 03 14:22:14.690082 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:22:14.690055    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da9b341_ead3_421e_95cf_ce7f1e1730c6.slice/crio-bf61f4e94415633d8f920f57ab53dc5a743e8af727270a7c1f7d5795ed6946c6 WatchSource:0}: Error finding container bf61f4e94415633d8f920f57ab53dc5a743e8af727270a7c1f7d5795ed6946c6: Status 404 returned error can't find the container with id bf61f4e94415633d8f920f57ab53dc5a743e8af727270a7c1f7d5795ed6946c6
Nov 03 14:22:14.692262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.692213    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:14.692356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.692294    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:14.692356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:14.692324    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:15.218671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:15.218637    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" event={"ID":"6da9b341-ead3-421e-95cf-ce7f1e1730c6","Type":"ContainerStarted","Data":"7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"}
Nov 03 14:22:15.219062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:15.218687    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" event={"ID":"6da9b341-ead3-421e-95cf-ce7f1e1730c6","Type":"ContainerStarted","Data":"bf61f4e94415633d8f920f57ab53dc5a743e8af727270a7c1f7d5795ed6946c6"}
Nov 03 14:22:15.294644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:15.294593    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podStartSLOduration=1.294576413 podStartE2EDuration="1.294576413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:15.280221111 +0000 UTC m=+613.330252331" watchObservedRunningTime="2026-11-03 14:22:15.294576413 +0000 UTC m=+613.344607596"
Nov 03 14:22:15.349708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:15.349675    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:15.354778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:15.354757    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:16.224389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:16.224358    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:16.225456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:16.225436    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:16.422794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:16.422766    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:16.644461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:16.644374    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:17.732460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:17.732426    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:17.971209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:17.971154    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:18.725956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:18.725894    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:19.562803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.562772    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:19.649906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.649877    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"]
Nov 03 14:22:19.653764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.653750    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.676334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.676312    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"]
Nov 03 14:22:19.685838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.685807    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2d0da2-dockercfg-fdb9z\""
Nov 03 14:22:19.685838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.685810    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2d0da2\""
Nov 03 14:22:19.748509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.748449    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5977w\" (UniqueName: \"kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w\") pod \"payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.748793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.748759    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.753310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.753278    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"]
Nov 03 14:22:19.759402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.759379    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:19.771554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.771521    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"]
Nov 03 14:22:19.787077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.787041    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:19.850155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.850079    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5977w\" (UniqueName: \"kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w\") pod \"payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.850155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.850116    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.850417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.850172    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lzn\" (UniqueName: \"kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn\") pod \"payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:19.850417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.850268    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:19.850875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.850854    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.871909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.871877    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5977w\" (UniqueName: \"kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w\") pod \"payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.951694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.951671    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lzn\" (UniqueName: \"kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn\") pod \"payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:19.951799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.951725    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:19.952338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.952322    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:19.967452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.967429    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:19.973263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:19.973241    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lzn\" (UniqueName: \"kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn\") pod \"payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:20.088405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:20.088376    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:20.266663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:20.266634    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"]
Nov 03 14:22:20.267426 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:22:20.267392    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc877e4_adb7_4022_92bf_3dacf7e50950.slice/crio-d7dc399d36e0605c6fc8589564da06b65f55f62541c499ac7df29d4c4381bcc7 WatchSource:0}: Error finding container d7dc399d36e0605c6fc8589564da06b65f55f62541c499ac7df29d4c4381bcc7: Status 404 returned error can't find the container with id d7dc399d36e0605c6fc8589564da06b65f55f62541c499ac7df29d4c4381bcc7
Nov 03 14:22:20.386348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:20.386324    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"]
Nov 03 14:22:20.386818 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:22:20.386788    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a92b45a_231f_4e27_a401_bed7dbf93732.slice/crio-49e5b9a6aef41f2224beb9cd8365953364c1eaa6be070a34d15a61f7bd727bf1 WatchSource:0}: Error finding container 49e5b9a6aef41f2224beb9cd8365953364c1eaa6be070a34d15a61f7bd727bf1: Status 404 returned error can't find the container with id 49e5b9a6aef41f2224beb9cd8365953364c1eaa6be070a34d15a61f7bd727bf1
Nov 03 14:22:20.743204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:20.743167    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:21.262192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.262156    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" event={"ID":"1bc877e4-adb7-4022-92bf-3dacf7e50950","Type":"ContainerStarted","Data":"5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407"}
Nov 03 14:22:21.262396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.262215    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" event={"ID":"1bc877e4-adb7-4022-92bf-3dacf7e50950","Type":"ContainerStarted","Data":"d7dc399d36e0605c6fc8589564da06b65f55f62541c499ac7df29d4c4381bcc7"}
Nov 03 14:22:21.262396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.262344    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:21.263728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.263702    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" event={"ID":"3a92b45a-231f-4e27-a401-bed7dbf93732","Type":"ContainerStarted","Data":"e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f"}
Nov 03 14:22:21.263876 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.263743    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" event={"ID":"3a92b45a-231f-4e27-a401-bed7dbf93732","Type":"ContainerStarted","Data":"49e5b9a6aef41f2224beb9cd8365953364c1eaa6be070a34d15a61f7bd727bf1"}
Nov 03 14:22:21.263876 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.263826    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:21.305557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.305518    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" podStartSLOduration=2.305505384 podStartE2EDuration="2.305505384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:21.304607368 +0000 UTC m=+619.354638551" watchObservedRunningTime="2026-11-03 14:22:21.305505384 +0000 UTC m=+619.355536567"
Nov 03 14:22:21.349925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:21.349878    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" podStartSLOduration=2.349865721 podStartE2EDuration="2.349865721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:21.348378626 +0000 UTC m=+619.398409810" watchObservedRunningTime="2026-11-03 14:22:21.349865721 +0000 UTC m=+619.399896904"
Nov 03 14:22:22.704954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:22.704898    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:23.921201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:23.921167    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:24.144225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:24.144183    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:24.711357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:24.711301    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:25.001449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:25.001365    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:22:26.747619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:26.747580    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:27.986262 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:22:27.986205    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:22:28.820369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:28.820336    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:30.782754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:30.782702    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:31.011989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:31.011951    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:31.237845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:31.237785    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:32.272837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:32.272805    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:22:32.273207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:32.272876    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:22:32.886602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:32.886565    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:34.893996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:34.893960    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:36.423122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:36.423080    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:36.740661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:36.740564    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:22:36.917186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:36.917156    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:37.820460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:37.808546    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:38.925774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:38.925693    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:40.141263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.139807    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"]
Nov 03 14:22:40.148125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.148072    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.165810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.165783    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"]
Nov 03 14:22:40.237246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237198    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237316    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237374    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237506    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237610    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237646    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237726    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhzn\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237795    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.237877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.237833    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339003    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339038    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339067    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhzn\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339108    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339141    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339192    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339427    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339508    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339570    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339663    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339817    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339918    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.339993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.339979    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.340124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.340048    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.341736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.341711    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.342067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.342044    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.360489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.360466    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.360922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.360902    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhzn\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.464598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.464566    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:40.798276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.798246    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"]
Nov 03 14:22:40.799701 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:22:40.799673    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c50b4c_b00e_4e6b_ac89_869bd3e4fde4.slice/crio-9471814d00b4f57c0336fe6568711da41af0f26ede56e2e7ea818221746febb1 WatchSource:0}: Error finding container 9471814d00b4f57c0336fe6568711da41af0f26ede56e2e7ea818221746febb1: Status 404 returned error can't find the container with id 9471814d00b4f57c0336fe6568711da41af0f26ede56e2e7ea818221746febb1
Nov 03 14:22:40.826972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.826927    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:40.827069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.827029    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:40.827110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.827073    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:40.915434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:40.906971    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:41.385207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:41.385171    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" event={"ID":"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4","Type":"ContainerStarted","Data":"38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab"}
Nov 03 14:22:41.385629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:41.385258    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" event={"ID":"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4","Type":"ContainerStarted","Data":"9471814d00b4f57c0336fe6568711da41af0f26ede56e2e7ea818221746febb1"}
Nov 03 14:22:41.432733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:41.432683    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podStartSLOduration=1.432668489 podStartE2EDuration="1.432668489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:41.432101772 +0000 UTC m=+639.482132954" watchObservedRunningTime="2026-11-03 14:22:41.432668489 +0000 UTC m=+639.482699671"
Nov 03 14:22:41.465594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:41.465563    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:41.467424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:41.467396    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:41.467574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:41.467440    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:42.465561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:42.465524    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:42.466066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:42.465577    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:42.955809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:42.955774    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:43.465659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:43.465626    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:43.466067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:43.465686    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:44.465560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:44.465521    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:44.465852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:44.465810    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:44.986647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:44.983814    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:45.465073 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:45.465040    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:45.465275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:45.465096    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:46.469215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:46.468676    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:46.469215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:46.468738    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:47.152069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:47.151992    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:47.267422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:47.267374    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:22:47.267621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:47.267445    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:22:47.466780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:47.466737    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Nov 03 14:22:47.466988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:47.466802    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Nov 03 14:22:48.004465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.004423    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:48.154045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.153999    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:22:48.484716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.484675    6481 generic.go:356] "Generic (PLEG): container finished" podID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerID="7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821" exitCode=0
Nov 03 14:22:48.484906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.484744    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" event={"ID":"6da9b341-ead3-421e-95cf-ce7f1e1730c6","Type":"ContainerDied","Data":"7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"}
Nov 03 14:22:48.485220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.485202    6481 scope.go:122] "RemoveContainer" containerID="7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"
Nov 03 14:22:48.486608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.486579    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:48.486710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.486663    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:48.486772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:48.486711    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:22:49.466538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:49.466500    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:22:49.466960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:49.466571    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:22:49.486962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:49.486929    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:49.492572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:49.492547    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" event={"ID":"6da9b341-ead3-421e-95cf-ce7f1e1730c6","Type":"ContainerStarted","Data":"51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03"}
Nov 03 14:22:50.349414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.349383    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:50.349629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.349559    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 14:22:50.349629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.349613    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 14:22:50.513379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.513323    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:22:50.514000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.513955    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:22:50.675818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.675790    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:50.676290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.676267    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:50.677112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:50.677095    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:22:51.350832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:51.350791    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 14:22:51.351157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:51.351120    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 14:22:51.461114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:51.461086    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:52.349774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:52.349734    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 14:22:52.350290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:52.349800    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 14:22:53.115281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.115246    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:53.247821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.247791    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"]
Nov 03 14:22:53.251260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.251243    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.272584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.272559    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"]
Nov 03 14:22:53.281747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.281722    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"auth-test-model-1ffaa0-kserve-self-signed-certs\""
Nov 03 14:22:53.281913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.281754    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"default-dockercfg-gk8b4\""
Nov 03 14:22:53.285516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.285495    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"openshift-service-ca.crt\""
Nov 03 14:22:53.285516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.285513    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-1ffaa0\"/\"kube-root-ca.crt\""
Nov 03 14:22:53.287008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.286985    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.287101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.287012    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.287101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.287041    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.287215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.287103    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.287215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.287158    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76sb\" (UniqueName: \"kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.287403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.287383    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.349350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.349317    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 14:22:53.349524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.349362    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 14:22:53.388168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388084    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388184    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388219    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388277    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388300    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388338    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76sb\" (UniqueName: \"kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388722    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388728    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.388962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.388937    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.390572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.390549    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.390723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.390704    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.425130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.424986    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:53.426322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.426296    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76sb\" (UniqueName: \"kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb\") pod \"auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") " pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.564527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.564489    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:53.654896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.650652    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:53.898177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:53.898154    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"]
Nov 03 14:22:53.899250 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:22:53.899209    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc109d21_9632_48ec_8f57_5e143137f3e9.slice/crio-b390290ae45a9cbd1413752d2a64ce98d1f82e0060304d9f119c2b9194ee37ab WatchSource:0}: Error finding container b390290ae45a9cbd1413752d2a64ce98d1f82e0060304d9f119c2b9194ee37ab: Status 404 returned error can't find the container with id b390290ae45a9cbd1413752d2a64ce98d1f82e0060304d9f119c2b9194ee37ab
Nov 03 14:22:54.349870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:54.349830    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Nov 03 14:22:54.350051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:54.349896    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Nov 03 14:22:54.567436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:54.567392    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" event={"ID":"dc109d21-9632-48ec-8f57-5e143137f3e9","Type":"ContainerStarted","Data":"3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65"}
Nov 03 14:22:54.567914 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:54.567447    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" event={"ID":"dc109d21-9632-48ec-8f57-5e143137f3e9","Type":"ContainerStarted","Data":"b390290ae45a9cbd1413752d2a64ce98d1f82e0060304d9f119c2b9194ee37ab"}
Nov 03 14:22:54.615252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:54.615136    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podStartSLOduration=1.615117331 podStartE2EDuration="1.615117331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:54.612154644 +0000 UTC m=+652.662185831" watchObservedRunningTime="2026-11-03 14:22:54.615117331 +0000 UTC m=+652.665148515"
Nov 03 14:22:55.354621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:55.354590    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:55.354953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:55.354918    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:55.355727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:55.355708    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:22:55.506371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:55.506333    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:55.565218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:55.565187    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:22:56.735117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:56.735074    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:57.513445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:57.513412    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:57.986146 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:22:57.986107    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:22:58.145549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:58.145501    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:22:59.408086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:59.408034    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:22:59.513678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:22:59.513647    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:01.503104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:01.503066    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:02.568705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:02.568648    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:03.515940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:03.515910    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:03.565284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:03.565254    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:23:03.578340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:03.578315    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:23:03.631830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:03.631805    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:23:04.306913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:04.306870    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:05.515964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:05.515912    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:07.528378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:07.528341    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:09.040476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.040442    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"]
Nov 03 14:23:09.043893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.043875    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.062929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.062900    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:23:09.072924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.072897    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:23:09.073211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.073166    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:23:09.073386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.073368    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:23:09.075630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.075609    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"]
Nov 03 14:23:09.204130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.204099    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rbf\" (UniqueName: \"kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.204312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.204164    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.204577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.204551    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.306222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.306146    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.306222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.306198    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rbf\" (UniqueName: \"kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.306427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.306223    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.306973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.306945    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.308550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.308528    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.328568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.328541    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rbf\" (UniqueName: \"kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf\") pod \"maas-api-revoke-sub-df9b138e-5lxpw\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.356193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.356171    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:09.503932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.503889    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:09.653310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.653283    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"]
Nov 03 14:23:09.654109 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:09.654081    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74870f1c_4c34_4eb5_95bd_e154dcf7c39a.slice/crio-a0adc4cbb14355962ee02a47e747ea1bc301f33c864d023ab71ec52938172c7e WatchSource:0}: Error finding container a0adc4cbb14355962ee02a47e747ea1bc301f33c864d023ab71ec52938172c7e: Status 404 returned error can't find the container with id a0adc4cbb14355962ee02a47e747ea1bc301f33c864d023ab71ec52938172c7e
Nov 03 14:23:09.669291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:09.669259    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw" event={"ID":"74870f1c-4c34-4eb5-95bd-e154dcf7c39a","Type":"ContainerStarted","Data":"a0adc4cbb14355962ee02a47e747ea1bc301f33c864d023ab71ec52938172c7e"}
Nov 03 14:23:10.677079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:10.677000    6481 generic.go:356] "Generic (PLEG): container finished" podID="74870f1c-4c34-4eb5-95bd-e154dcf7c39a" containerID="96bc72f7acc1d84c5716525ce133995be4cea3056c9e74ee3b08f706c1a504b1" exitCode=0
Nov 03 14:23:10.677472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:10.677086    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw" event={"ID":"74870f1c-4c34-4eb5-95bd-e154dcf7c39a","Type":"ContainerDied","Data":"96bc72f7acc1d84c5716525ce133995be4cea3056c9e74ee3b08f706c1a504b1"}
Nov 03 14:23:11.470635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.470603    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:11.826736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.826707    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:11.830289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.830268    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca\") pod \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") "
Nov 03 14:23:11.830385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.830372    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf\" (UniqueName: \"kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf\") pod \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") "
Nov 03 14:23:11.830440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.830428    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca\") pod \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\" (UID: \"74870f1c-4c34-4eb5-95bd-e154dcf7c39a\") "
Nov 03 14:23:11.830605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.830584    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca" pod "74870f1c-4c34-4eb5-95bd-e154dcf7c39a" (UID: "74870f1c-4c34-4eb5-95bd-e154dcf7c39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:23:11.830752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.830727    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:11.832475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.832453    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca" pod "74870f1c-4c34-4eb5-95bd-e154dcf7c39a" (UID: "74870f1c-4c34-4eb5-95bd-e154dcf7c39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:23:11.832551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.832471    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf" pod "74870f1c-4c34-4eb5-95bd-e154dcf7c39a" (UID: "74870f1c-4c34-4eb5-95bd-e154dcf7c39a"). InnerVolumeSpecName "kube-api-access-c6rbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:23:11.931491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.931459    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6rbf\" (UniqueName: \"kubernetes.io/projected/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-kube-api-access-c6rbf\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:11.931491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:11.931487    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74870f1c-4c34-4eb5-95bd-e154dcf7c39a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:12.688493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:12.688465    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"
Nov 03 14:23:12.688709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:12.688503    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw" event={"ID":"74870f1c-4c34-4eb5-95bd-e154dcf7c39a","Type":"ContainerDied","Data":"a0adc4cbb14355962ee02a47e747ea1bc301f33c864d023ab71ec52938172c7e"}
Nov 03 14:23:12.688709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:12.688548    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0adc4cbb14355962ee02a47e747ea1bc301f33c864d023ab71ec52938172c7e"
Nov 03 14:23:13.417108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:13.417071    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:13.799038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:13.798959    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:23:14.723355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.723311    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"]
Nov 03 14:23:14.724820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.724793    6481 cpu_manager.go:414] "containerMap: removing container" podUID="74870f1c-4c34-4eb5-95bd-e154dcf7c39a" containerName="revoke-keys"
Nov 03 14:23:14.725002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.724986    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74870f1c-4c34-4eb5-95bd-e154dcf7c39a" containerName="revoke-keys"
Nov 03 14:23:14.725305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.725288    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="74870f1c-4c34-4eb5-95bd-e154dcf7c39a" containerName="revoke-keys"
Nov 03 14:23:14.731255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.729656    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.769646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.768572    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-1ffaa0-openshift-default-dockercfg-k7w6w\""
Nov 03 14:23:14.769646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.768590    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"]
Nov 03 14:23:14.792030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.791983    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792121    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25lfv\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792182    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792214    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792473    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792533    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792566    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792604    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.792660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.792630    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.837853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.837705    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894391    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25lfv\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894483    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894519    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894550    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894599    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894636    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894674    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894699    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.895260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.894832    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.896200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.896111    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.896200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.896166    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.896520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.896495    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.896612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.896537    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.896676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.896626    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.898968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.898947    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.899373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.899348    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.966269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.954549    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:14.966269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:14.955211    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25lfv\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:15.053669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.053584    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:15.328893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.328809    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:15.445195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.437402    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"]
Nov 03 14:23:15.445464 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:15.445209    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506204d2_8aa7_4d8c_9c70_4abfc29836c6.slice/crio-29d8cf5fb0f0a6d2827eda2f988cae504303f36b13451112880f8ad424e047d0 WatchSource:0}: Error finding container 29d8cf5fb0f0a6d2827eda2f988cae504303f36b13451112880f8ad424e047d0: Status 404 returned error can't find the container with id 29d8cf5fb0f0a6d2827eda2f988cae504303f36b13451112880f8ad424e047d0
Nov 03 14:23:15.467384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.467342    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:15.467488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.467455    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:15.467544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.467526    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:15.557591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.557566    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:15.710080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.710048    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" event={"ID":"506204d2-8aa7-4d8c-9c70-4abfc29836c6","Type":"ContainerStarted","Data":"d46616747896d7f41d4a0b2d196adb6cb16cb451e0c6c11018e01ea2197b2927"}
Nov 03 14:23:15.710277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.710099    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" event={"ID":"506204d2-8aa7-4d8c-9c70-4abfc29836c6","Type":"ContainerStarted","Data":"29d8cf5fb0f0a6d2827eda2f988cae504303f36b13451112880f8ad424e047d0"}
Nov 03 14:23:15.758686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:15.758630    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podStartSLOduration=1.758610826 podStartE2EDuration="1.758610826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:15.755595633 +0000 UTC m=+673.805626817" watchObservedRunningTime="2026-11-03 14:23:15.758610826 +0000 UTC m=+673.808642012"
Nov 03 14:23:16.054265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:16.054170    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:16.055905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:16.055875    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 14:23:16.056042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:16.055938    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 14:23:17.054359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:17.054323    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 14:23:17.054799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:17.054376    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 14:23:17.430773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:17.430746    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:18.054707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:18.054671    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 14:23:18.055096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:18.054733    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 14:23:18.331687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:18.331594    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:18.893714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:18.893687    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:19.055093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:19.055056    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 14:23:19.055520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:19.055104    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 14:23:19.123682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:19.123650    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:19.695452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:19.695409    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:20.054582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:20.054498    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 14:23:20.054582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:20.054549    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 14:23:20.728420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:20.728253    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:20.988777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:20.981852    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:21.055074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:21.055032    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 03 14:23:21.055288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:21.055106    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 03 14:23:21.809049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:21.809004    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:22.091150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:22.091063    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:22.091546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:22.091523    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:22.092814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:22.092792    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:23:23.814618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:23.814585    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:25.737842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.737802    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"]
Nov 03 14:23:25.742015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.741980    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:25.762849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.762825    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"]
Nov 03 14:23:25.773173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.773150    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"auth-test-model-2d0da2-kserve-self-signed-certs\""
Nov 03 14:23:25.773323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.773170    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"openshift-service-ca.crt\""
Nov 03 14:23:25.773323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.773169    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"default-dockercfg-fpknl\""
Nov 03 14:23:25.773323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.773212    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"kube-root-ca.crt\""
Nov 03 14:23:25.909893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.909859    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:25.918222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.918197    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:25.918321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.918309    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:25.918377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.918363    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:25.918423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.918410    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:25.918548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.918532    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:25.918595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:25.918582    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.020351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.020194    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.020351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.020267    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.020588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.020405    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.020588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.020458    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.020588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.020533    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.020743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.020603    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.021044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.021017    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.021644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.021619    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.021929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.021902    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.025348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.025327    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.027485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.027460    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.043792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.043760    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq\") pod \"auth-test-model-2d0da2-kserve-599ccc648f-f6sk2\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.058260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.058238    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:26.130492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.130463    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:26.361897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.361871    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"]
Nov 03 14:23:26.363296 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:26.363269    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61c4cd10_f303_4b2f_8208_41a0cb736312.slice/crio-4494bd0497de5d15ab12b551693862944f2b450b000553eb78ace05da729e94b WatchSource:0}: Error finding container 4494bd0497de5d15ab12b551693862944f2b450b000553eb78ace05da729e94b: Status 404 returned error can't find the container with id 4494bd0497de5d15ab12b551693862944f2b450b000553eb78ace05da729e94b
Nov 03 14:23:26.809659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.809625    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" event={"ID":"61c4cd10-f303-4b2f-8208-41a0cb736312","Type":"ContainerStarted","Data":"f34bde8e56935a6554b7148c33a6bf31b8c90ee2c9cd0b4be6fe9f0bc4116741"}
Nov 03 14:23:26.810043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.809671    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" event={"ID":"61c4cd10-f303-4b2f-8208-41a0cb736312","Type":"ContainerStarted","Data":"4494bd0497de5d15ab12b551693862944f2b450b000553eb78ace05da729e94b"}
Nov 03 14:23:26.856286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:26.856220    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podStartSLOduration=1.8562017119999998 podStartE2EDuration="1.856201712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:26.854262589 +0000 UTC m=+684.904293773" watchObservedRunningTime="2026-11-03 14:23:26.856201712 +0000 UTC m=+684.906232892"
Nov 03 14:23:27.059412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:27.059385    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:27.987327 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:23:27.987295    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:23:27.999601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:27.999576    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:30.081463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:30.081431    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:31.011331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:31.011297    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:23:32.135432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:32.135400    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:34.145334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:34.145297    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:34.385479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:34.385446    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:35.106021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:35.105991    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:35.126145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:35.126118    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"]
Nov 03 14:23:35.183518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:35.183486    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"]
Nov 03 14:23:35.393070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:35.392993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:35.696960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:35.696921    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:36.100296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:36.086732    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:36.121561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:36.121529    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:36.556466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:36.556435    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:36.883438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:36.883355    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:23:37.712474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.712435    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"]
Nov 03 14:23:37.716484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.716467    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.739041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.739014    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"]
Nov 03 14:23:37.747853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.747819    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:23:37.747853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.747819    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:23:37.748036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.747912    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:23:37.748036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.747968    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:23:37.777504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.777468    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.777636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.777599    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.777697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.777674    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.878210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.878179    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.878421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.878241    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.878421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.878347    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.879039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.879014    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.880684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.880661    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:37.901898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:37.901869    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5\") pod \"maas-api-revoke-sub-8b46debb-gpp8f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:38.030447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.030370    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:38.228493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.228458    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:38.335557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.335531    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"]
Nov 03 14:23:38.336268 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:38.336217    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0360855e_b5e0_4bbe_b754_0dc3a4d34a0f.slice/crio-58e476acf213d6acafb89facf8c19a40950283e042246eacffd5ba820699c818 WatchSource:0}: Error finding container 58e476acf213d6acafb89facf8c19a40950283e042246eacffd5ba820699c818: Status 404 returned error can't find the container with id 58e476acf213d6acafb89facf8c19a40950283e042246eacffd5ba820699c818
Nov 03 14:23:38.610325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.610274    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:38.885676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.885602    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f" event={"ID":"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f","Type":"ContainerStarted","Data":"d7a5890b5d0dce4bc5435aaddcf5c80b256f32d555afd4f77540128c1fbb98c3"}
Nov 03 14:23:38.885676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.885649    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f" event={"ID":"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f","Type":"ContainerStarted","Data":"58e476acf213d6acafb89facf8c19a40950283e042246eacffd5ba820699c818"}
Nov 03 14:23:38.930301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:38.930255    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f" podStartSLOduration=1.930242154 podStartE2EDuration="1.930242154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:38.929334794 +0000 UTC m=+696.979365978" watchObservedRunningTime="2026-11-03 14:23:38.930242154 +0000 UTC m=+696.980273328"
Nov 03 14:23:39.893624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:39.893591    6481 generic.go:356] "Generic (PLEG): container finished" podID="0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" containerID="d7a5890b5d0dce4bc5435aaddcf5c80b256f32d555afd4f77540128c1fbb98c3" exitCode=0
Nov 03 14:23:39.894028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:39.893659    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f" event={"ID":"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f","Type":"ContainerDied","Data":"d7a5890b5d0dce4bc5435aaddcf5c80b256f32d555afd4f77540128c1fbb98c3"}
Nov 03 14:23:40.127028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.126984    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerName="payload-processing" containerID="cri-o://ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280" gracePeriod=25
Nov 03 14:23:40.184034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.183956    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerName="payload-pre-processing" containerID="cri-o://97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f" gracePeriod=25
Nov 03 14:23:40.341392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.341354    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"]
Nov 03 14:23:40.346001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.345980    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.358125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.357733    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"]
Nov 03 14:23:40.362777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.362753    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.373160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.373133    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"]
Nov 03 14:23:40.387822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.387791    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"]
Nov 03 14:23:40.407464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407432    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407466    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.407668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407553    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407593    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfqb\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407638    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407681    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407743    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407766    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407785    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.407891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407811    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vzw\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.407891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407874    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.408013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407911    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.408013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.407930    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.408112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.408061    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.408168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.408115    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.408168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.408147    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.408301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.408217    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.408301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.408295    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509165    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509207    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509289    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509315    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509387    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509411    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509451    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509497    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfqb\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509528    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509610    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509675    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509705    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509730    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.509905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509747    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509769    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vzw\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.509905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509866    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509908    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.509938    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510007    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.510182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510044    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.510182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510058    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510077    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.510499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510271    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.510499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510370    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510431    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.510499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510456    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510509    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.510841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.510817    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.512025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.512000    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.512413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.512394    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.512846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.512824    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.513003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.512986    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.532242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.532191    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.532355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.532304    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.532445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.532425    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vzw\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.533061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.533044    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfqb\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:40.633053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.633016    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:40.665166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.665140    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:40.679435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:40.679416    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:41.029663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.024438    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"]
Nov 03 14:23:41.029663 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:41.025742    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a590d8_6a27_465e_942b_b6b116dd9cdd.slice/crio-73092d8d2dfc01c1b9338c96327ce49521fcc638fe639e0c5cb8112c7bf408fc WatchSource:0}: Error finding container 73092d8d2dfc01c1b9338c96327ce49521fcc638fe639e0c5cb8112c7bf408fc: Status 404 returned error can't find the container with id 73092d8d2dfc01c1b9338c96327ce49521fcc638fe639e0c5cb8112c7bf408fc
Nov 03 14:23:41.031696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.030522    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:41.031696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.030608    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:41.031696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.030649    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:41.033656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.033614    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"]
Nov 03 14:23:41.036781 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:41.036752    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519203e3_e560_4068_b4da_58a90d6d4564.slice/crio-6e6ad610efd0fd183ed2aa6cf2b9618cd7c2aa74a519611a36823ce088647d72 WatchSource:0}: Error finding container 6e6ad610efd0fd183ed2aa6cf2b9618cd7c2aa74a519611a36823ce088647d72: Status 404 returned error can't find the container with id 6e6ad610efd0fd183ed2aa6cf2b9618cd7c2aa74a519611a36823ce088647d72
Nov 03 14:23:41.042952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.042889    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:41.043120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.043102    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:41.043266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.043248    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:23:41.148021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.147938    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:41.232381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.227819    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca\") pod \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") "
Nov 03 14:23:41.232381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.227994    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca\") pod \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") "
Nov 03 14:23:41.232381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.228062    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5\") pod \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\" (UID: \"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f\") "
Nov 03 14:23:41.232381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.230429    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca" pod "0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" (UID: "0360855e-b5e0-4bbe-b754-0dc3a4d34a0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:23:41.278974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.278929    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:41.316341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.306606    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5" pod "0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" (UID: "0360855e-b5e0-4bbe-b754-0dc3a4d34a0f"). InnerVolumeSpecName "kube-api-access-fgpv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:23:41.330256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.324351    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca" pod "0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" (UID: "0360855e-b5e0-4bbe-b754-0dc3a4d34a0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:23:41.390254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.386407    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:41.390254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.386446    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgpv5\" (UniqueName: \"kubernetes.io/projected/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f-kube-api-access-fgpv5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:41.949399 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.949359    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f" event={"ID":"0360855e-b5e0-4bbe-b754-0dc3a4d34a0f","Type":"ContainerDied","Data":"58e476acf213d6acafb89facf8c19a40950283e042246eacffd5ba820699c818"}
Nov 03 14:23:41.949801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.949778    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58e476acf213d6acafb89facf8c19a40950283e042246eacffd5ba820699c818"
Nov 03 14:23:41.950024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.950009    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"
Nov 03 14:23:41.958980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.957642    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" event={"ID":"519203e3-e560-4068-b4da-58a90d6d4564","Type":"ContainerStarted","Data":"7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a"}
Nov 03 14:23:41.958980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.957703    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" event={"ID":"519203e3-e560-4068-b4da-58a90d6d4564","Type":"ContainerStarted","Data":"6e6ad610efd0fd183ed2aa6cf2b9618cd7c2aa74a519611a36823ce088647d72"}
Nov 03 14:23:41.960734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.960711    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" event={"ID":"58a590d8-6a27-465e-942b-b6b116dd9cdd","Type":"ContainerStarted","Data":"9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28"}
Nov 03 14:23:41.960845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:41.960748    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" event={"ID":"58a590d8-6a27-465e-942b-b6b116dd9cdd","Type":"ContainerStarted","Data":"73092d8d2dfc01c1b9338c96327ce49521fcc638fe639e0c5cb8112c7bf408fc"}
Nov 03 14:23:42.025263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.015961    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podStartSLOduration=2.015941996 podStartE2EDuration="2.015941996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:42.014697201 +0000 UTC m=+700.064728414" watchObservedRunningTime="2026-11-03 14:23:42.015941996 +0000 UTC m=+700.065973183"
Nov 03 14:23:42.097165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.097114    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podStartSLOduration=2.097098826 podStartE2EDuration="2.097098826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:42.093293896 +0000 UTC m=+700.143325080" watchObservedRunningTime="2026-11-03 14:23:42.097098826 +0000 UTC m=+700.147130009"
Nov 03 14:23:42.666169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.666138    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:42.668489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.668456    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 14:23:42.668615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.668521    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 14:23:42.679803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.679779    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:42.680576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.680555    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 14:23:42.680627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.680593    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 14:23:42.991276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:42.991195    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:43.665743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:43.665709    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 14:23:43.666268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:43.665760    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 14:23:43.686252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:43.684200    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 14:23:43.686252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:43.684298    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 14:23:44.666607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:44.666560    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 14:23:44.667088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:44.667062    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 14:23:44.680602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:44.680573    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 14:23:44.680704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:44.680615    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 14:23:45.282474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.282386    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:45.547319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.546771    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:45.558311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.558271    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:23:45.678095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.678061    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 14:23:45.678742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.678713    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 14:23:45.689756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.689733    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 14:23:45.689966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:45.689945    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 14:23:46.551255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:46.550685    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:23:46.666300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:46.666141    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 14:23:46.666300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:46.666190    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 14:23:46.684106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:46.684076    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 14:23:46.684666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:46.684635    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 14:23:47.503394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:47.503353    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:47.665947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:47.665907    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Nov 03 14:23:47.666138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:47.665976    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Nov 03 14:23:47.680408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:47.680366    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Nov 03 14:23:47.680571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:47.680434    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Nov 03 14:23:48.904102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:48.904070    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:48.905841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:48.905667    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:48.908794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:48.908774    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:23:48.996251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:48.995102    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:48.996251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:48.995635    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:48.996648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:48.996623    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:23:49.017993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:49.017959    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:49.657196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:49.657161    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:52.019166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.019110    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:52.131052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.131015    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"]
Nov 03 14:23:52.131629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.131610    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" containerName="revoke-keys"
Nov 03 14:23:52.131743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.131630    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" containerName="revoke-keys"
Nov 03 14:23:52.131812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.131797    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" containerName="revoke-keys"
Nov 03 14:23:52.135111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.135092    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.157523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.157497    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"]
Nov 03 14:23:52.165471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.165436    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:23:52.165592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.165569    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:23:52.165675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.165632    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:23:52.165737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.165676    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:23:52.249471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.249440    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.249471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.249471    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.249680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.249662    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf85\" (UniqueName: \"kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.350523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.350441    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf85\" (UniqueName: \"kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.350670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.350549    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.350670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.350577    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.351313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.351290    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.352978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.352951    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.375064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.375038    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf85\" (UniqueName: \"kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85\") pod \"maas-api-revoke-sub-e7556663-kn22k\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.449022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.448992    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:52.749929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:52.749893    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"]
Nov 03 14:23:52.750291 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:23:52.750268    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e23ad60_fc64_49dd_98ca_2e65a72d9882.slice/crio-b689a209860e6cdab8ece57d51b111b3b00238238e7c040c7f06741be4b9e3c0 WatchSource:0}: Error finding container b689a209860e6cdab8ece57d51b111b3b00238238e7c040c7f06741be4b9e3c0: Status 404 returned error can't find the container with id b689a209860e6cdab8ece57d51b111b3b00238238e7c040c7f06741be4b9e3c0
Nov 03 14:23:53.124676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:53.124637    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k" event={"ID":"8e23ad60-fc64-49dd-98ca-2e65a72d9882","Type":"ContainerStarted","Data":"3ab50dfb1c0ebb9030707aca15501651acdbc274a8a46a13422af6f293da9761"}
Nov 03 14:23:53.125133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:53.124691    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k" event={"ID":"8e23ad60-fc64-49dd-98ca-2e65a72d9882","Type":"ContainerStarted","Data":"b689a209860e6cdab8ece57d51b111b3b00238238e7c040c7f06741be4b9e3c0"}
Nov 03 14:23:53.169611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:53.169562    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k" podStartSLOduration=1.1695488410000001 podStartE2EDuration="1.169548841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:53.167178639 +0000 UTC m=+711.217209822" watchObservedRunningTime="2026-11-03 14:23:53.169548841 +0000 UTC m=+711.219580070"
Nov 03 14:23:53.324472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:53.324397    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:53.836952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:53.836916    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:23:54.122742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:54.122672    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:54.133723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:54.133687    6481 generic.go:356] "Generic (PLEG): container finished" podID="8e23ad60-fc64-49dd-98ca-2e65a72d9882" containerID="3ab50dfb1c0ebb9030707aca15501651acdbc274a8a46a13422af6f293da9761" exitCode=0
Nov 03 14:23:54.134120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:54.133774    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k" event={"ID":"8e23ad60-fc64-49dd-98ca-2e65a72d9882","Type":"ContainerDied","Data":"3ab50dfb1c0ebb9030707aca15501651acdbc274a8a46a13422af6f293da9761"}
Nov 03 14:23:54.354251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:54.351498    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:54.839669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:54.839631    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:55.411249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:55.410962    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:55.818668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:55.818363    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:23:56.082304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.069925    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:56.597033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.596604    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:23:56.624264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.621131    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:56.751701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.751665    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85\" (UniqueName: \"kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85\") pod \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") "
Nov 03 14:23:56.761487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.761457    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca\") pod \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") "
Nov 03 14:23:56.761749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.761728    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca\") pod \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\" (UID: \"8e23ad60-fc64-49dd-98ca-2e65a72d9882\") "
Nov 03 14:23:56.766810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.766777    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Nov 03 14:23:56.783741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.783707    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca" pod "8e23ad60-fc64-49dd-98ca-2e65a72d9882" (UID: "8e23ad60-fc64-49dd-98ca-2e65a72d9882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:23:56.807595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.807554    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85" pod "8e23ad60-fc64-49dd-98ca-2e65a72d9882" (UID: "8e23ad60-fc64-49dd-98ca-2e65a72d9882"). InnerVolumeSpecName "kube-api-access-tgf85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:23:56.830116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.830073    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca" pod "8e23ad60-fc64-49dd-98ca-2e65a72d9882" (UID: "8e23ad60-fc64-49dd-98ca-2e65a72d9882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:23:56.859882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.859752    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:56.884252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.876749    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:56.912355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.912302    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e23ad60-fc64-49dd-98ca-2e65a72d9882-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:56.912554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.912537    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgf85\" (UniqueName: \"kubernetes.io/projected/8e23ad60-fc64-49dd-98ca-2e65a72d9882-kube-api-access-tgf85\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:23:56.912660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:56.905117    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:23:57.107261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:57.107065    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:23:57.600138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:57.600055    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"
Nov 03 14:23:57.600138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:57.600076    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k" event={"ID":"8e23ad60-fc64-49dd-98ca-2e65a72d9882","Type":"ContainerDied","Data":"b689a209860e6cdab8ece57d51b111b3b00238238e7c040c7f06741be4b9e3c0"}
Nov 03 14:23:57.600699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:57.600146    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b689a209860e6cdab8ece57d51b111b3b00238238e7c040c7f06741be4b9e3c0"
Nov 03 14:23:57.985981 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:23:57.985944    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:23:58.994481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:23:58.994448    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:00.446272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.446216    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:00.448147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.448129    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:24:00.469339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.469319    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume\") pod \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") "
Nov 03 14:24:00.469456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.469366    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4\" (UniqueName: \"kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4\") pod \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\" (UID: \"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6\") "
Nov 03 14:24:00.469669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.469648    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume" pod "4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" (UID: "4f55d01a-43dc-40d3-9729-ec4b9e6b57b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:00.469889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.469872    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:00.471648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.471628    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4" pod "4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" (UID: "4f55d01a-43dc-40d3-9729-ec4b9e6b57b6"). InnerVolumeSpecName "kube-api-access-h7tq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:00.476007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.475988    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:24:00.571124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.571089    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume\") pod \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") "
Nov 03 14:24:00.571333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.571309    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk\" (UniqueName: \"kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk\") pod \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\" (UID: \"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0\") "
Nov 03 14:24:00.571489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.571465    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume" pod "f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" (UID: "f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:00.571769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.571747    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7tq4\" (UniqueName: \"kubernetes.io/projected/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6-kube-api-access-h7tq4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:00.571836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.571775    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:00.573442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.573418    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk" pod "f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" (UID: "f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0"). InnerVolumeSpecName "kube-api-access-8jfdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:00.619662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.619626    6481 generic.go:356] "Generic (PLEG): container finished" podID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerID="ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280" exitCode=0
Nov 03 14:24:00.619797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.619670    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"
Nov 03 14:24:00.619797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.619723    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" event={"ID":"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6","Type":"ContainerDied","Data":"ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280"}
Nov 03 14:24:00.619797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.619773    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b" event={"ID":"4f55d01a-43dc-40d3-9729-ec4b9e6b57b6","Type":"ContainerDied","Data":"bd1cc8a4b8de79d3f6835545a75a220c2b1577ab684426f812d1a24870aa29c9"}
Nov 03 14:24:00.619921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.619805    6481 scope.go:122] "RemoveContainer" containerID="ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280"
Nov 03 14:24:00.621258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.621206    6481 generic.go:356] "Generic (PLEG): container finished" podID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerID="97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f" exitCode=0
Nov 03 14:24:00.621373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.621268    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" event={"ID":"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0","Type":"ContainerDied","Data":"97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f"}
Nov 03 14:24:00.621373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.621281    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"
Nov 03 14:24:00.621373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.621326    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd" event={"ID":"f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0","Type":"ContainerDied","Data":"9aab589d44567f9d0e29ef83466b93aba6713368d2b5724bf63c6fa8a12b24a0"}
Nov 03 14:24:00.630133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.630114    6481 scope.go:122] "RemoveContainer" containerID="ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280"
Nov 03 14:24:00.630500 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:24:00.630477    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280\": container with ID starting with ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280 not found: ID does not exist" containerID="ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280"
Nov 03 14:24:00.630615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.630539    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280"} err="failed to get container status \"ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280\": rpc error: code = NotFound desc = could not find container \"ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280\": container with ID starting with ddb0c91922b5633cd85312ad75b3f328d3cd9ff666e4a3a20b8a0676b41af280 not found: ID does not exist"
Nov 03 14:24:00.630615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.630565    6481 scope.go:122] "RemoveContainer" containerID="97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f"
Nov 03 14:24:00.639184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.639167    6481 scope.go:122] "RemoveContainer" containerID="97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f"
Nov 03 14:24:00.639439 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:24:00.639418    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f\": container with ID starting with 97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f not found: ID does not exist" containerID="97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f"
Nov 03 14:24:00.639508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.639448    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f"} err="failed to get container status \"97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f\": rpc error: code = NotFound desc = could not find container \"97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f\": container with ID starting with 97d29960a0348d749ec2cd7baf6620ec5dc1369f13dcc544f605aea72237073f not found: ID does not exist"
Nov 03 14:24:00.672532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.672510    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfdk\" (UniqueName: \"kubernetes.io/projected/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0-kube-api-access-8jfdk\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:00.693579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.693561    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:00.694865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.694845    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"]
Nov 03 14:24:00.700623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.700603    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4429aa-648f6bdb7-dhpmd"]
Nov 03 14:24:00.761888 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.761861    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"]
Nov 03 14:24:00.771942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.771918    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4429aa-7d69cbff69-vvx2b"]
Nov 03 14:24:00.923012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:00.922928    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:01.971139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:01.971107    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:02.073540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.073498    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" path="/var/lib/kubelet/pods/4f55d01a-43dc-40d3-9729-ec4b9e6b57b6/volumes"
Nov 03 14:24:02.074219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.074198    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" path="/var/lib/kubelet/pods/f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0/volumes"
Nov 03 14:24:02.433828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.433789    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:02.511375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511341    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"]
Nov 03 14:24:02.511921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511904    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerName="payload-pre-processing"
Nov 03 14:24:02.511965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511922    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerName="payload-pre-processing"
Nov 03 14:24:02.511965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511932    6481 cpu_manager.go:414] "containerMap: removing container" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerName="payload-processing"
Nov 03 14:24:02.511965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511937    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerName="payload-processing"
Nov 03 14:24:02.512058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511975    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8e23ad60-fc64-49dd-98ca-2e65a72d9882" containerName="revoke-keys"
Nov 03 14:24:02.512058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.511980    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e23ad60-fc64-49dd-98ca-2e65a72d9882" containerName="revoke-keys"
Nov 03 14:24:02.512115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.512065    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e23ad60-fc64-49dd-98ca-2e65a72d9882" containerName="revoke-keys"
Nov 03 14:24:02.512115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.512074    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9c1f6f2-a8b2-42d7-adbe-0a3e2e52dbf0" containerName="payload-pre-processing"
Nov 03 14:24:02.512115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.512082    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f55d01a-43dc-40d3-9729-ec4b9e6b57b6" containerName="payload-processing"
Nov 03 14:24:02.516793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.516777    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.537594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.537572    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"]
Nov 03 14:24:02.546534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.546511    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-e759e5\""
Nov 03 14:24:02.546534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.546527    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-e759e5-dockercfg-5mxhh\""
Nov 03 14:24:02.601947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.601914    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.601947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.601958    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8lc\" (UniqueName: \"kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc\") pod \"payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.684301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.682832    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:02.706109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.706073    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.706408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.706141    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8lc\" (UniqueName: \"kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc\") pod \"payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.707976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.707943    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.729350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.729321    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8lc\" (UniqueName: \"kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc\") pod \"payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:02.830197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:02.830169    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:03.132256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:03.132211    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"]
Nov 03 14:24:03.132973 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:03.132945    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5c42db_c9e0_49c0_b008_33d595ea3c78.slice/crio-50ede6932b37308e0c756d14409f06d09753e6b05a02f6fbf1a322542f2c5260 WatchSource:0}: Error finding container 50ede6932b37308e0c756d14409f06d09753e6b05a02f6fbf1a322542f2c5260: Status 404 returned error can't find the container with id 50ede6932b37308e0c756d14409f06d09753e6b05a02f6fbf1a322542f2c5260
Nov 03 14:24:03.660800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:03.660763    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" event={"ID":"ef5c42db-c9e0-49c0-b008-33d595ea3c78","Type":"ContainerStarted","Data":"e8da05c80a84a07f5bd00f250fe09b6ae1427d84467f677637f8106df66453c2"}
Nov 03 14:24:03.660948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:03.660819    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" event={"ID":"ef5c42db-c9e0-49c0-b008-33d595ea3c78","Type":"ContainerStarted","Data":"50ede6932b37308e0c756d14409f06d09753e6b05a02f6fbf1a322542f2c5260"}
Nov 03 14:24:03.660948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:03.660880    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:03.705221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:03.705165    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" podStartSLOduration=1.705151364 podStartE2EDuration="1.705151364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:03.703977348 +0000 UTC m=+721.754008532" watchObservedRunningTime="2026-11-03 14:24:03.705151364 +0000 UTC m=+721.755182547"
Nov 03 14:24:03.925574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:03.925493    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:05.906179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:05.906141    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:06.914826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.914794    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"]
Nov 03 14:24:06.925216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.925181    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:06.942465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.942440    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"]
Nov 03 14:24:06.950513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.950487    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:24:06.950659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.950540    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:24:06.950659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.950640    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:24:06.954406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:06.954387    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:24:07.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.060510    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4mb\" (UniqueName: \"kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.060774    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.060865    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.161964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.161932    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4mb\" (UniqueName: \"kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.162143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.162019    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.162143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.162056    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.162663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.162639    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.164430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.164402    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.183590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.183534    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4mb\" (UniqueName: \"kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb\") pod \"maas-api-revoke-sub-676e120e-twb8b\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.239113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.239086    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:07.561345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.560933    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"]
Nov 03 14:24:07.685476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:07.685440    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b" event={"ID":"83022cb0-9424-4629-bb1f-464c772af24a","Type":"ContainerStarted","Data":"17c36fbbdba7b1db0f31938ca74175b096776ad7f7cd0d77c7abed41edf2bce3"}
Nov 03 14:24:08.009174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:08.009145    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:08.692864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:08.692830    6481 generic.go:356] "Generic (PLEG): container finished" podID="83022cb0-9424-4629-bb1f-464c772af24a" containerID="1b3c0e559857fb7800ff6ab386611f653a258f11652801668d0631521ac67305" exitCode=0
Nov 03 14:24:08.693030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:08.692881    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b" event={"ID":"83022cb0-9424-4629-bb1f-464c772af24a","Type":"ContainerDied","Data":"1b3c0e559857fb7800ff6ab386611f653a258f11652801668d0631521ac67305"}
Nov 03 14:24:09.844277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.844255    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:09.896347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.896314    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca\") pod \"83022cb0-9424-4629-bb1f-464c772af24a\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") "
Nov 03 14:24:09.896482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.896407    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb\" (UniqueName: \"kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb\") pod \"83022cb0-9424-4629-bb1f-464c772af24a\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") "
Nov 03 14:24:09.896565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.896533    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca\") pod \"83022cb0-9424-4629-bb1f-464c772af24a\" (UID: \"83022cb0-9424-4629-bb1f-464c772af24a\") "
Nov 03 14:24:09.897043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.897016    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca" pod "83022cb0-9424-4629-bb1f-464c772af24a" (UID: "83022cb0-9424-4629-bb1f-464c772af24a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:09.898513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.898492    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb" pod "83022cb0-9424-4629-bb1f-464c772af24a" (UID: "83022cb0-9424-4629-bb1f-464c772af24a"). InnerVolumeSpecName "kube-api-access-wf4mb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:09.898513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.898502    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca" pod "83022cb0-9424-4629-bb1f-464c772af24a" (UID: "83022cb0-9424-4629-bb1f-464c772af24a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:24:09.997984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.997905    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4mb\" (UniqueName: \"kubernetes.io/projected/83022cb0-9424-4629-bb1f-464c772af24a-kube-api-access-wf4mb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:09.997984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.997931    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:09.997984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:09.997942    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83022cb0-9424-4629-bb1f-464c772af24a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:10.130961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.130927    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:10.242371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.242337    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"]
Nov 03 14:24:10.242884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.242872    6481 cpu_manager.go:414] "containerMap: removing container" podUID="83022cb0-9424-4629-bb1f-464c772af24a" containerName="revoke-keys"
Nov 03 14:24:10.242926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.242886    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83022cb0-9424-4629-bb1f-464c772af24a" containerName="revoke-keys"
Nov 03 14:24:10.242978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.242969    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="83022cb0-9424-4629-bb1f-464c772af24a" containerName="revoke-keys"
Nov 03 14:24:10.246258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.246238    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.268746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.268723    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"]
Nov 03 14:24:10.272431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.272415    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.282961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.282941    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"]
Nov 03 14:24:10.305315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305283    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305336    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305373    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305447    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305520    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305556    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305597    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjwwx\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305640    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.305805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.305715    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.323389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.323356    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"]
Nov 03 14:24:10.407137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407106    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407168    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.407323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407216    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407257    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407334    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407425    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407460    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.407636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407503    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.407706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407683    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407680    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.407761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407733    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.407865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407764    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.407865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407763    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.407865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407846    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.408016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407884    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.408016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407914    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.408016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407944    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjrq\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.408016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.407982    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjwwx\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.408163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.408140    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.408199 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.408165    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.408288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.408267    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.408375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.408355    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.408792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.408775    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.409557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.409539    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.409833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.409806    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.434469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.434447    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.434561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.434515    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjwwx\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.510020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.509961    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510081    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510117    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510160    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510189    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510215    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510287    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510310    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjrq\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510398    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510580    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510648    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510694    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.510807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.510720    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.511022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.511000    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.512281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.512263    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.512603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.512588    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.536241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.536211    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.536328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.536219    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjrq\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.560165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.560140    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:10.588287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.588266    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:10.708169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.708140    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"
Nov 03 14:24:10.708318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.708141    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b" event={"ID":"83022cb0-9424-4629-bb1f-464c772af24a","Type":"ContainerDied","Data":"17c36fbbdba7b1db0f31938ca74175b096776ad7f7cd0d77c7abed41edf2bce3"}
Nov 03 14:24:10.708318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.708282    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c36fbbdba7b1db0f31938ca74175b096776ad7f7cd0d77c7abed41edf2bce3"
Nov 03 14:24:10.910724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.910699    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"]
Nov 03 14:24:10.911292 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:10.911270    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058cc99f_47ec_4285_a436_2095e30669e2.slice/crio-b39e524a5025ecaa7d157d2457642c76f4c4275a586fc58fbc74af60c523325b WatchSource:0}: Error finding container b39e524a5025ecaa7d157d2457642c76f4c4275a586fc58fbc74af60c523325b: Status 404 returned error can't find the container with id b39e524a5025ecaa7d157d2457642c76f4c4275a586fc58fbc74af60c523325b
Nov 03 14:24:10.914030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.913999    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:24:10.914123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.914059    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:24:10.914123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.914087    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:24:10.949204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.949179    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"]
Nov 03 14:24:10.958546 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:10.958516    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154bc5be_af4d_4236_b2af_58c839a2eedf.slice/crio-967c60d64afa20aa7df040cd3de5d3518d163159678a4e90518fa577b70659e7 WatchSource:0}: Error finding container 967c60d64afa20aa7df040cd3de5d3518d163159678a4e90518fa577b70659e7: Status 404 returned error can't find the container with id 967c60d64afa20aa7df040cd3de5d3518d163159678a4e90518fa577b70659e7
Nov 03 14:24:10.960916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.960886    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:24:10.961012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.960950    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:24:10.961012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:10.960979    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:24:11.757330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:11.757291    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" event={"ID":"058cc99f-47ec-4285-a436-2095e30669e2","Type":"ContainerStarted","Data":"161b28006d76bbc748ecacd2a97e5e93de7cea321ee066179e917c54fb08754c"}
Nov 03 14:24:11.757517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:11.757348    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" event={"ID":"058cc99f-47ec-4285-a436-2095e30669e2","Type":"ContainerStarted","Data":"b39e524a5025ecaa7d157d2457642c76f4c4275a586fc58fbc74af60c523325b"}
Nov 03 14:24:11.759750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:11.759726    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" event={"ID":"154bc5be-af4d-4236-b2af-58c839a2eedf","Type":"ContainerStarted","Data":"b491b14049d7496512685a0e617acb78b8e19b7144006b418647e9a2e10e33ef"}
Nov 03 14:24:11.759886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:11.759765    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" event={"ID":"154bc5be-af4d-4236-b2af-58c839a2eedf","Type":"ContainerStarted","Data":"967c60d64afa20aa7df040cd3de5d3518d163159678a4e90518fa577b70659e7"}
Nov 03 14:24:11.855039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:11.854976    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podStartSLOduration=1.854958605 podStartE2EDuration="1.854958605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:11.852518059 +0000 UTC m=+729.902549466" watchObservedRunningTime="2026-11-03 14:24:11.854958605 +0000 UTC m=+729.904989790"
Nov 03 14:24:11.921253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:11.901064    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podStartSLOduration=1.901047434 podStartE2EDuration="1.901047434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:11.898586017 +0000 UTC m=+729.948617201" watchObservedRunningTime="2026-11-03 14:24:11.901047434 +0000 UTC m=+729.951078619"
Nov 03 14:24:12.254261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.241081    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:12.555311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.554361    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:12.562295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.561300    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:12.565903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.565870    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 14:24:12.566049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.565922    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 14:24:12.589421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.589399    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:12.590608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.590587    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 14:24:12.590711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.590628    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 14:24:12.770576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:12.770545    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:13.377014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:13.376932    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:13.560778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:13.560740    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 14:24:13.560976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:13.560810    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 14:24:13.589558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:13.589527    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 14:24:13.589701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:13.589579    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 14:24:13.834253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:13.828999    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:14.562129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:14.562082    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 14:24:14.562688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:14.562146    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 14:24:14.589378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:14.589348    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 14:24:14.589545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:14.589400    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 14:24:14.616735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:14.616702    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:14.669505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:14.669473    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:24:15.561433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:15.561401    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 14:24:15.561680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:15.561657    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 14:24:15.589219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:15.589192    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 14:24:15.589664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:15.589635    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 14:24:16.561511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:16.561468    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 14:24:16.561683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:16.561534    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 14:24:16.590007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:16.589977    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 14:24:16.590398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:16.590030    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 14:24:16.681898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:16.681866    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:16.924703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:16.923572    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:17.561035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:17.561002    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Nov 03 14:24:17.561217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:17.561057    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Nov 03 14:24:17.589303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:17.589263    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Nov 03 14:24:17.589455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:17.589323    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Nov 03 14:24:17.905610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:17.905516    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:24:19.116363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.116330    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:19.494596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.494563    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:19.615254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.611334    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:24:19.615254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.611427    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:19.630375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.630345    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:24:19.630593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.630567    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:19.781474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.781396    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:19.781827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.781799    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:19.782499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.782481    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:24:19.948805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.948773    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"]
Nov 03 14:24:19.952702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.952677    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:19.970512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.970486    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:24:19.970685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.970617    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:24:19.970775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.970746    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:24:19.970889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.970872    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:24:19.973933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:19.973908    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"]
Nov 03 14:24:20.003343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.000972    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.003343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.001095    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.003343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.001430    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbtr\" (UniqueName: \"kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.014820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.014794    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:20.015527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.015496    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:20.016424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.016374    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:24:20.102435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.102372    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.102642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.102624    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbtr\" (UniqueName: \"kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.102757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.102735    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.103052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.103034    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.104949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.104928    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.128351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.128326    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbtr\" (UniqueName: \"kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr\") pod \"maas-api-revoke-sub-2636454a-qs9zg\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.267845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.267809    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:20.573657 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:20.573628    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43129c35_2401_4252_b4ce_fe074b85ede5.slice/crio-b84b2323ea789b61455142ebab840b8600d0a9c8f943cc5810f46724101603fc WatchSource:0}: Error finding container b84b2323ea789b61455142ebab840b8600d0a9c8f943cc5810f46724101603fc: Status 404 returned error can't find the container with id b84b2323ea789b61455142ebab840b8600d0a9c8f943cc5810f46724101603fc
Nov 03 14:24:20.573787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.573725    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"]
Nov 03 14:24:20.938497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.938468    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg" event={"ID":"43129c35-2401-4252-b4ce-fe074b85ede5","Type":"ContainerStarted","Data":"e39bd9cb789481dc91f159dac65bc1d4b8eb11be5c79265cf170d36be938ff4b"}
Nov 03 14:24:20.938626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.938525    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg" event={"ID":"43129c35-2401-4252-b4ce-fe074b85ede5","Type":"ContainerStarted","Data":"b84b2323ea789b61455142ebab840b8600d0a9c8f943cc5810f46724101603fc"}
Nov 03 14:24:20.982572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:20.982527    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg" podStartSLOduration=1.982513625 podStartE2EDuration="1.982513625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:20.981327683 +0000 UTC m=+739.031358866" watchObservedRunningTime="2026-11-03 14:24:20.982513625 +0000 UTC m=+739.032544801"
Nov 03 14:24:21.155622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:21.155587    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:21.945960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:21.945926    6481 generic.go:356] "Generic (PLEG): container finished" podID="43129c35-2401-4252-b4ce-fe074b85ede5" containerID="e39bd9cb789481dc91f159dac65bc1d4b8eb11be5c79265cf170d36be938ff4b" exitCode=0
Nov 03 14:24:21.946161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:21.946013    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg" event={"ID":"43129c35-2401-4252-b4ce-fe074b85ede5","Type":"ContainerDied","Data":"e39bd9cb789481dc91f159dac65bc1d4b8eb11be5c79265cf170d36be938ff4b"}
Nov 03 14:24:23.096685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.096658    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:23.139914 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.139818    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr\" (UniqueName: \"kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr\") pod \"43129c35-2401-4252-b4ce-fe074b85ede5\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") "
Nov 03 14:24:23.140415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.140042    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca\") pod \"43129c35-2401-4252-b4ce-fe074b85ede5\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") "
Nov 03 14:24:23.140415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.140087    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca\") pod \"43129c35-2401-4252-b4ce-fe074b85ede5\" (UID: \"43129c35-2401-4252-b4ce-fe074b85ede5\") "
Nov 03 14:24:23.142304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.141290    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca" pod "43129c35-2401-4252-b4ce-fe074b85ede5" (UID: "43129c35-2401-4252-b4ce-fe074b85ede5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:23.143048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.143025    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca" pod "43129c35-2401-4252-b4ce-fe074b85ede5" (UID: "43129c35-2401-4252-b4ce-fe074b85ede5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:24:23.145706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.145679    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr" pod "43129c35-2401-4252-b4ce-fe074b85ede5" (UID: "43129c35-2401-4252-b4ce-fe074b85ede5"). InnerVolumeSpecName "kube-api-access-dtbtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:23.228258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.228135    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:23.241818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.241782    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:23.241818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.241817    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43129c35-2401-4252-b4ce-fe074b85ede5-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:23.241994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.241831    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtbtr\" (UniqueName: \"kubernetes.io/projected/43129c35-2401-4252-b4ce-fe074b85ede5-kube-api-access-dtbtr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:23.960188 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.960159    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"
Nov 03 14:24:23.960188 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.960172    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg" event={"ID":"43129c35-2401-4252-b4ce-fe074b85ede5","Type":"ContainerDied","Data":"b84b2323ea789b61455142ebab840b8600d0a9c8f943cc5810f46724101603fc"}
Nov 03 14:24:23.960416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:23.960214    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b84b2323ea789b61455142ebab840b8600d0a9c8f943cc5810f46724101603fc"
Nov 03 14:24:24.597835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:24.597796    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"]
Nov 03 14:24:24.865307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:24.865207    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:25.090610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:25.090541    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:25.327493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:25.327448    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:27.186579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:27.186544    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:27.423067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:27.423033    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:27.990364 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:24:27.990327    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:24:29.602996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:29.602945    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk" podUID="844db57e-253a-4573-978d-48222f1e2cc9" containerName="payload-processing" containerID="cri-o://427646f4b5dd4d6604f7471d3f24de32cf205b8611f5a9a0de49e0618b061f72" gracePeriod=25
Nov 03 14:24:29.669619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:29.669585    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:29.925648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:29.923329    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:30.573579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.544848    6481 generic.go:356] "Generic (PLEG): container finished" podID="844db57e-253a-4573-978d-48222f1e2cc9" containerID="427646f4b5dd4d6604f7471d3f24de32cf205b8611f5a9a0de49e0618b061f72" exitCode=0
Nov 03 14:24:30.627142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.547673    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk" event={"ID":"844db57e-253a-4573-978d-48222f1e2cc9","Type":"ContainerDied","Data":"427646f4b5dd4d6604f7471d3f24de32cf205b8611f5a9a0de49e0618b061f72"}
Nov 03 14:24:30.640571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.640545    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:24:30.842315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.831024    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume\") pod \"844db57e-253a-4573-978d-48222f1e2cc9\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") "
Nov 03 14:24:30.880333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.880301    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4\" (UniqueName: \"kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4\") pod \"844db57e-253a-4573-978d-48222f1e2cc9\" (UID: \"844db57e-253a-4573-978d-48222f1e2cc9\") "
Nov 03 14:24:30.899732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.886462    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume" pod "844db57e-253a-4573-978d-48222f1e2cc9" (UID: "844db57e-253a-4573-978d-48222f1e2cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:30.922521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.922454    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4" pod "844db57e-253a-4573-978d-48222f1e2cc9" (UID: "844db57e-253a-4573-978d-48222f1e2cc9"). InnerVolumeSpecName "kube-api-access-pbvv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:30.987717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.987676    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvv4\" (UniqueName: \"kubernetes.io/projected/844db57e-253a-4573-978d-48222f1e2cc9-kube-api-access-pbvv4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:30.987961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:30.987942    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/844db57e-253a-4573-978d-48222f1e2cc9-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:31.643283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:31.643256    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"
Nov 03 14:24:31.643283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:31.643269    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk" event={"ID":"844db57e-253a-4573-978d-48222f1e2cc9","Type":"ContainerDied","Data":"80a9e4422ad2878753c24e341c763ba74daa9c2bcb385cb6db757e71afdd9108"}
Nov 03 14:24:31.643907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:31.643339    6481 scope.go:122] "RemoveContainer" containerID="427646f4b5dd4d6604f7471d3f24de32cf205b8611f5a9a0de49e0618b061f72"
Nov 03 14:24:31.701138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:31.701109    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"]
Nov 03 14:24:31.714172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:31.714148    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0a0ec6-8595fdb499-pvwkk"]
Nov 03 14:24:31.932972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:31.932931    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:32.068520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:32.068488    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844db57e-253a-4573-978d-48222f1e2cc9" path="/var/lib/kubelet/pods/844db57e-253a-4573-978d-48222f1e2cc9/volumes"
Nov 03 14:24:33.862924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:33.862893    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:34.250790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:34.250759    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:35.119359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.119324    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"]
Nov 03 14:24:35.119891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.119876    6481 cpu_manager.go:414] "containerMap: removing container" podUID="844db57e-253a-4573-978d-48222f1e2cc9" containerName="payload-processing"
Nov 03 14:24:35.119939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.119892    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844db57e-253a-4573-978d-48222f1e2cc9" containerName="payload-processing"
Nov 03 14:24:35.119939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.119905    6481 cpu_manager.go:414] "containerMap: removing container" podUID="43129c35-2401-4252-b4ce-fe074b85ede5" containerName="revoke-keys"
Nov 03 14:24:35.119939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.119912    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43129c35-2401-4252-b4ce-fe074b85ede5" containerName="revoke-keys"
Nov 03 14:24:35.120044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.120005    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="844db57e-253a-4573-978d-48222f1e2cc9" containerName="payload-processing"
Nov 03 14:24:35.120044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.120012    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="43129c35-2401-4252-b4ce-fe074b85ede5" containerName="revoke-keys"
Nov 03 14:24:35.124748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.124732    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.144941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.144915    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"]
Nov 03 14:24:35.153820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.153789    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:24:35.153917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.153840    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:24:35.153917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.153887    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:24:35.154004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.153933    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:24:35.259567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.259539    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.259747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.259655    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.259793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.259749    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ws4\" (UniqueName: \"kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.360491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.360462    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.360654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.360550    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.360654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.360646    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ws4\" (UniqueName: \"kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.361201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.361175    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.362825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.362803    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.381919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.381863    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ws4\" (UniqueName: \"kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4\") pod \"maas-api-revoke-sub-883e8847-d8pl6\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.448475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.448437    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:35.808942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.808916    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"]
Nov 03 14:24:35.809894 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:35.809868    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd64381_bb15_40c4_a9dc_a99760b00cff.slice/crio-9e07c36915a6936581f0a625dee022a4250f1a3ba9ceb0369a02d0b5dce7c6aa WatchSource:0}: Error finding container 9e07c36915a6936581f0a625dee022a4250f1a3ba9ceb0369a02d0b5dce7c6aa: Status 404 returned error can't find the container with id 9e07c36915a6936581f0a625dee022a4250f1a3ba9ceb0369a02d0b5dce7c6aa
Nov 03 14:24:35.922500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:35.922470    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:36.676903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:36.676876    6481 generic.go:356] "Generic (PLEG): container finished" podID="0fd64381-bb15-40c4-a9dc-a99760b00cff" containerID="f04a6136464f23ef625c3a1097f737ecee5000a2d40308e84374ed04c5cd5980" exitCode=0
Nov 03 14:24:36.677225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:36.676918    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6" event={"ID":"0fd64381-bb15-40c4-a9dc-a99760b00cff","Type":"ContainerDied","Data":"f04a6136464f23ef625c3a1097f737ecee5000a2d40308e84374ed04c5cd5980"}
Nov 03 14:24:36.677225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:36.676953    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6" event={"ID":"0fd64381-bb15-40c4-a9dc-a99760b00cff","Type":"ContainerStarted","Data":"9e07c36915a6936581f0a625dee022a4250f1a3ba9ceb0369a02d0b5dce7c6aa"}
Nov 03 14:24:36.781841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:36.781802    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:24:37.908790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:37.908750    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:37.948344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:37.948263    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:38.177253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.162716    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca\") pod \"0fd64381-bb15-40c4-a9dc-a99760b00cff\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") "
Nov 03 14:24:38.177253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.166335    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4\" (UniqueName: \"kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4\") pod \"0fd64381-bb15-40c4-a9dc-a99760b00cff\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") "
Nov 03 14:24:38.177253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.166479    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca\") pod \"0fd64381-bb15-40c4-a9dc-a99760b00cff\" (UID: \"0fd64381-bb15-40c4-a9dc-a99760b00cff\") "
Nov 03 14:24:38.177253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.167640    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca" pod "0fd64381-bb15-40c4-a9dc-a99760b00cff" (UID: "0fd64381-bb15-40c4-a9dc-a99760b00cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:38.341921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.239706    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca" pod "0fd64381-bb15-40c4-a9dc-a99760b00cff" (UID: "0fd64381-bb15-40c4-a9dc-a99760b00cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:24:38.341921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.247457    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4" pod "0fd64381-bb15-40c4-a9dc-a99760b00cff" (UID: "0fd64381-bb15-40c4-a9dc-a99760b00cff"). InnerVolumeSpecName "kube-api-access-h8ws4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:38.341921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.273115    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:38.341921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.273147    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ws4\" (UniqueName: \"kubernetes.io/projected/0fd64381-bb15-40c4-a9dc-a99760b00cff-kube-api-access-h8ws4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:38.341921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.273171    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd64381-bb15-40c4-a9dc-a99760b00cff-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:38.895254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.875256    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6" event={"ID":"0fd64381-bb15-40c4-a9dc-a99760b00cff","Type":"ContainerDied","Data":"9e07c36915a6936581f0a625dee022a4250f1a3ba9ceb0369a02d0b5dce7c6aa"}
Nov 03 14:24:38.895254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.875326    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e07c36915a6936581f0a625dee022a4250f1a3ba9ceb0369a02d0b5dce7c6aa"
Nov 03 14:24:38.895254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:38.875398    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"
Nov 03 14:24:39.122334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:39.097922    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:39.421255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:39.419332    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:24:39.421255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:39.419435    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:40.849064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.849034    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"]
Nov 03 14:24:40.849724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.849707    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0fd64381-bb15-40c4-a9dc-a99760b00cff" containerName="revoke-keys"
Nov 03 14:24:40.849794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.849725    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd64381-bb15-40c4-a9dc-a99760b00cff" containerName="revoke-keys"
Nov 03 14:24:40.849893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.849881    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd64381-bb15-40c4-a9dc-a99760b00cff" containerName="revoke-keys"
Nov 03 14:24:40.853640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.853621    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:40.873492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.873472    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"]
Nov 03 14:24:40.883892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.883867    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:24:40.883892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.883877    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:24:40.884076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.883948    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:24:40.884076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.884006    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:24:40.899427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.899399    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:40.899588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.899573    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtpq\" (UniqueName: \"kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:40.899641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:40.899608    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.000670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.000640    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.000807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.000790    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtpq\" (UniqueName: \"kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.000866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.000835    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.001642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.001620    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.003009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.002990    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.024003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.023979    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtpq\" (UniqueName: \"kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq\") pod \"maas-api-revoke-sub-19f4127c-9zx9n\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.033692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.033242    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:41.167281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.167215    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:41.258946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.258882    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:41.489283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.489244    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"]
Nov 03 14:24:41.490175 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:41.490141    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9005fd_a7bc_404d_9bb3_db7cb9dea027.slice/crio-0474b5745178d4eef9dfa747dd29f8c7761297ab14322fcdd5339dea429a1a9c WatchSource:0}: Error finding container 0474b5745178d4eef9dfa747dd29f8c7761297ab14322fcdd5339dea429a1a9c: Status 404 returned error can't find the container with id 0474b5745178d4eef9dfa747dd29f8c7761297ab14322fcdd5339dea429a1a9c
Nov 03 14:24:41.898282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.898224    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n" event={"ID":"6d9005fd-a7bc-404d-9bb3-db7cb9dea027","Type":"ContainerStarted","Data":"4e84a9653f9106570b38ada542ac63604d6562962b079151514031574b40ee53"}
Nov 03 14:24:41.898621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.898302    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n" event={"ID":"6d9005fd-a7bc-404d-9bb3-db7cb9dea027","Type":"ContainerStarted","Data":"0474b5745178d4eef9dfa747dd29f8c7761297ab14322fcdd5339dea429a1a9c"}
Nov 03 14:24:41.939957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:41.939913    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n" podStartSLOduration=1.939900162 podStartE2EDuration="1.939900162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:41.938859312 +0000 UTC m=+759.988890488" watchObservedRunningTime="2026-11-03 14:24:41.939900162 +0000 UTC m=+759.989931342"
Nov 03 14:24:42.233821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:42.233778    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:42.483005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:42.482960    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:42.905605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:42.905574    6481 generic.go:356] "Generic (PLEG): container finished" podID="6d9005fd-a7bc-404d-9bb3-db7cb9dea027" containerID="4e84a9653f9106570b38ada542ac63604d6562962b079151514031574b40ee53" exitCode=0
Nov 03 14:24:42.906010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:42.905658    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n" event={"ID":"6d9005fd-a7bc-404d-9bb3-db7cb9dea027","Type":"ContainerDied","Data":"4e84a9653f9106570b38ada542ac63604d6562962b079151514031574b40ee53"}
Nov 03 14:24:43.136597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:43.136562    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:43.369941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:43.369912    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:43.426709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:43.426675    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"]
Nov 03 14:24:43.426949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:43.426924    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy" containerID="cri-o://de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6" gracePeriod=30
Nov 03 14:24:43.604156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:43.604078    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:44.064052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.064029    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:44.140624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.140588    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca\") pod \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") "
Nov 03 14:24:44.140839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.140818    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq\" (UniqueName: \"kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq\") pod \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") "
Nov 03 14:24:44.140919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.140870    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca\") pod \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\" (UID: \"6d9005fd-a7bc-404d-9bb3-db7cb9dea027\") "
Nov 03 14:24:44.141012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.140987    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca" pod "6d9005fd-a7bc-404d-9bb3-db7cb9dea027" (UID: "6d9005fd-a7bc-404d-9bb3-db7cb9dea027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:44.141458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.141433    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:44.143700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.143662    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq" pod "6d9005fd-a7bc-404d-9bb3-db7cb9dea027" (UID: "6d9005fd-a7bc-404d-9bb3-db7cb9dea027"). InnerVolumeSpecName "kube-api-access-dqtpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:44.152003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.151979    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca" pod "6d9005fd-a7bc-404d-9bb3-db7cb9dea027" (UID: "6d9005fd-a7bc-404d-9bb3-db7cb9dea027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:24:44.243212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.243145    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqtpq\" (UniqueName: \"kubernetes.io/projected/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-kube-api-access-dqtpq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:44.243212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.243172    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9005fd-a7bc-404d-9bb3-db7cb9dea027-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:44.924700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.924669    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"
Nov 03 14:24:44.924922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.924696    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n" event={"ID":"6d9005fd-a7bc-404d-9bb3-db7cb9dea027","Type":"ContainerDied","Data":"0474b5745178d4eef9dfa747dd29f8c7761297ab14322fcdd5339dea429a1a9c"}
Nov 03 14:24:44.924922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:44.924893    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0474b5745178d4eef9dfa747dd29f8c7761297ab14322fcdd5339dea429a1a9c"
Nov 03 14:24:45.390026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:45.389949    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:45.612947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:45.612904    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:45.920633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:45.920600    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:46.242834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:46.242755    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:47.515894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:47.515860    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:48.682082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.682058    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:24:48.831618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831534    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.831787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831626    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.831787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831649    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.831787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831687    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.831965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831810    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.831965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831823    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:24:48.831965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831895    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.831965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831923    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.832181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.831994    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.832181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832078    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:48.832181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832103    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket\") pod \"913318d5-bb9b-407e-8ba0-77f611c482a6\" (UID: \"913318d5-bb9b-407e-8ba0-77f611c482a6\") "
Nov 03 14:24:48.832378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832185    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:24:48.832436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832375    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:24:48.832486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832434    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:24:48.832741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832701    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.832741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832723    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/913318d5-bb9b-407e-8ba0-77f611c482a6-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.832741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832738    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.832921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832751    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.832921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.832760    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.834019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.833993    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:48.834289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.834269    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:24:48.834450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.834430    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:24:48.834564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.834543    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4" pod "913318d5-bb9b-407e-8ba0-77f611c482a6" (UID: "913318d5-bb9b-407e-8ba0-77f611c482a6"). InnerVolumeSpecName "kube-api-access-xk5h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:48.934081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.934052    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk5h4\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-kube-api-access-xk5h4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.934081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.934078    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.934274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.934089    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.934274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.934097    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/913318d5-bb9b-407e-8ba0-77f611c482a6-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:48.963331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.963299    6481 generic.go:356] "Generic (PLEG): container finished" podID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerID="de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6" exitCode=0
Nov 03 14:24:48.963468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.963347    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" event={"ID":"913318d5-bb9b-407e-8ba0-77f611c482a6","Type":"ContainerDied","Data":"de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6"}
Nov 03 14:24:48.963468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.963357    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"
Nov 03 14:24:48.963468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.963396    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs" event={"ID":"913318d5-bb9b-407e-8ba0-77f611c482a6","Type":"ContainerDied","Data":"3a5e20a07f42ce930f3562d1544780732b954c0eba570e1420614e2325b34ea2"}
Nov 03 14:24:48.963468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.963419    6481 scope.go:122] "RemoveContainer" containerID="de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6"
Nov 03 14:24:48.973882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.973862    6481 scope.go:122] "RemoveContainer" containerID="9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"
Nov 03 14:24:48.983194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.983170    6481 scope.go:122] "RemoveContainer" containerID="de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6"
Nov 03 14:24:48.983525 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:24:48.983496    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6\": container with ID starting with de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6 not found: ID does not exist" containerID="de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6"
Nov 03 14:24:48.983632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.983536    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6"} err="failed to get container status \"de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6\": rpc error: code = NotFound desc = could not find container \"de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6\": container with ID starting with de18e2eb64e0e3fa0ace9ade8c5b16c9eaaa376a15f4c83fb9ffbf4463ad2ea6 not found: ID does not exist"
Nov 03 14:24:48.983632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.983553    6481 scope.go:122] "RemoveContainer" containerID="9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"
Nov 03 14:24:48.983922 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:24:48.983884    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6\": container with ID starting with 9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6 not found: ID does not exist" containerID="9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"
Nov 03 14:24:48.983990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:48.983926    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6"} err="failed to get container status \"9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6\": rpc error: code = NotFound desc = could not find container \"9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6\": container with ID starting with 9ee6d8939e944a0f07c9855dbf870168393179cc8aa03e98620e9d5ce2b339b6 not found: ID does not exist"
Nov 03 14:24:49.022689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.022657    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"]
Nov 03 14:24:49.035083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.035053    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-0a0ec6-openshift-default-c4dff6869-p6qfs"]
Nov 03 14:24:49.237392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.237356    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:49.376320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.376287    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"]
Nov 03 14:24:49.376869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.376852    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6d9005fd-a7bc-404d-9bb3-db7cb9dea027" containerName="revoke-keys"
Nov 03 14:24:49.376869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.376870    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9005fd-a7bc-404d-9bb3-db7cb9dea027" containerName="revoke-keys"
Nov 03 14:24:49.376986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.376885    6481 cpu_manager.go:414] "containerMap: removing container" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy"
Nov 03 14:24:49.376986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.376890    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy"
Nov 03 14:24:49.377048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.376997    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy"
Nov 03 14:24:49.377048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.377005    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9005fd-a7bc-404d-9bb3-db7cb9dea027" containerName="revoke-keys"
Nov 03 14:24:49.377048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.377013    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy"
Nov 03 14:24:49.381759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.381740    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.407209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.403051    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:24:49.407209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.405412    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:24:49.407209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.405637    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:24:49.407209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.405866    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:24:49.407209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.407109    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"]
Nov 03 14:24:49.461382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.461348    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:49.543470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.543340    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.543645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.543612    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.543718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.543650    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bt7l\" (UniqueName: \"kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.644756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.644723    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.644756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.644758    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bt7l\" (UniqueName: \"kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.645020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.644830    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.645600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.645577    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.647192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.647168    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.687082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.687049    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bt7l\" (UniqueName: \"kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l\") pod \"maas-api-revoke-sub-1f46ed72-jrzjq\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.696747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.696722    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:49.836607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:49.836575    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:50.166262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:50.164931    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"]
Nov 03 14:24:50.265250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:50.254642    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" path="/var/lib/kubelet/pods/913318d5-bb9b-407e-8ba0-77f611c482a6/volumes"
Nov 03 14:24:50.855643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:50.855531    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:51.050978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.050349    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:51.283429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.283373    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:51.426958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.426847    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:51.701256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.664295    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"]
Nov 03 14:24:51.701256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.673302    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:51.856423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.856316    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:51.896334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.869200    6481 cpu_manager.go:414] "containerMap: removing container" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy"
Nov 03 14:24:51.896577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.896558    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="913318d5-bb9b-407e-8ba0-77f611c482a6" containerName="istio-proxy"
Nov 03 14:24:51.968254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:51.887401    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 14:24:52.070250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.058243    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:52.073458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.059653    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"]
Nov 03 14:24:52.086352 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.086319    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"]
Nov 03 14:24:52.087560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.087521    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:52.150312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.149384    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:52.205401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.205366    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.267884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.217404    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5l5\" (UniqueName: \"kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5\") pod \"payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.313254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.292696    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:24:52.313254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.229331    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.313254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.296110    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-331c9e\""
Nov 03 14:24:52.318304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.229259    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:24:52.423311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.415966    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq" event={"ID":"2651061b-aedc-49a6-a2c0-aa645a74a7c6","Type":"ContainerStarted","Data":"c596ac848ff6281ba781fa90fcaf48924752917b94743d5c26994bb3afcc2378"}
Nov 03 14:24:52.423311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.416030    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq" event={"ID":"2651061b-aedc-49a6-a2c0-aa645a74a7c6","Type":"ContainerStarted","Data":"e725205961a251b729a648744ec0c2a49b435624ef028dd7ba89e992a7a04492"}
Nov 03 14:24:52.423311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.416096    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:24:52.423311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.416139    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:24:52.423311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.416161    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"]
Nov 03 14:24:52.448251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.446183    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.473252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.470409    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-331c9e-dockercfg-rsxmw\""
Nov 03 14:24:52.575255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.572568    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.575255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.572882    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2fnb\" (UniqueName: \"kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb\") pod \"payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.575255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.572969    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5l5\" (UniqueName: \"kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5\") pod \"payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.575255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.573067    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.575255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.573960    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.608473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.608441    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5l5\" (UniqueName: \"kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5\") pod \"payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.676191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.676085    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.683456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.683424    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2fnb\" (UniqueName: \"kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb\") pod \"payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.684113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.677216    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.806263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.799009    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2fnb\" (UniqueName: \"kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb\") pod \"payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:52.815261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.799740    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:52.829273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.829020    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:52.883695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.881355    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:52.883695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.882161    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:24:52.977450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.977361    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:52.978064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:52.978040    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:24:53.035616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.035590    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:53.090986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.090575    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:24:53.198927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.197456    6481 generic.go:356] "Generic (PLEG): container finished" podID="2651061b-aedc-49a6-a2c0-aa645a74a7c6" containerID="c596ac848ff6281ba781fa90fcaf48924752917b94743d5c26994bb3afcc2378" exitCode=0
Nov 03 14:24:53.198927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.198251    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq" event={"ID":"2651061b-aedc-49a6-a2c0-aa645a74a7c6","Type":"ContainerDied","Data":"c596ac848ff6281ba781fa90fcaf48924752917b94743d5c26994bb3afcc2378"}
Nov 03 14:24:53.224389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.224348    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:53.289530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.283319    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:24:53.290614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.290315    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:53.291996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.290845    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:24:53.336937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.334523    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:24:53.483489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.481290    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:24:53.491270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.486331    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:24:53.491270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.487419    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:24:53.507021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.505928    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:24:53.598939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.598576    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:24:53.796336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.796302    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:53.815054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.815024    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l\" (UniqueName: \"kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l\") pod \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") "
Nov 03 14:24:53.815213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.815194    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca\") pod \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") "
Nov 03 14:24:53.815365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.815346    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca\") pod \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\" (UID: \"2651061b-aedc-49a6-a2c0-aa645a74a7c6\") "
Nov 03 14:24:53.815568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.815546    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca" pod "2651061b-aedc-49a6-a2c0-aa645a74a7c6" (UID: "2651061b-aedc-49a6-a2c0-aa645a74a7c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:24:53.815786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.815768    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:53.817103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.817083    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l" pod "2651061b-aedc-49a6-a2c0-aa645a74a7c6" (UID: "2651061b-aedc-49a6-a2c0-aa645a74a7c6"). InnerVolumeSpecName "kube-api-access-4bt7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:24:53.817223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.817210    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca" pod "2651061b-aedc-49a6-a2c0-aa645a74a7c6" (UID: "2651061b-aedc-49a6-a2c0-aa645a74a7c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:24:53.853543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.853458    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"]
Nov 03 14:24:53.865884 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:53.865857    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e30eee_5488_49a1_ad18_9b2ca9bd7224.slice/crio-4ecabe8b85b16b0707e26f109440ec62b3593c3183edc2afb527f762e2a94cfc WatchSource:0}: Error finding container 4ecabe8b85b16b0707e26f109440ec62b3593c3183edc2afb527f762e2a94cfc: Status 404 returned error can't find the container with id 4ecabe8b85b16b0707e26f109440ec62b3593c3183edc2afb527f762e2a94cfc
Nov 03 14:24:53.906147 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:53.906107    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f26cfa4_d3ab_4973_bfe8_bd9173f524d5.slice/crio-f1d01790d7b6cf946bed6e0d2cc1af21eaed0b20ec28791ec18926ba0ec75e37 WatchSource:0}: Error finding container f1d01790d7b6cf946bed6e0d2cc1af21eaed0b20ec28791ec18926ba0ec75e37: Status 404 returned error can't find the container with id f1d01790d7b6cf946bed6e0d2cc1af21eaed0b20ec28791ec18926ba0ec75e37
Nov 03 14:24:53.912695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.912654    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"]
Nov 03 14:24:53.916485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.916462    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2651061b-aedc-49a6-a2c0-aa645a74a7c6-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:53.916485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:53.916484    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bt7l\" (UniqueName: \"kubernetes.io/projected/2651061b-aedc-49a6-a2c0-aa645a74a7c6-kube-api-access-4bt7l\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:24:54.216253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.209455    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq" event={"ID":"2651061b-aedc-49a6-a2c0-aa645a74a7c6","Type":"ContainerDied","Data":"e725205961a251b729a648744ec0c2a49b435624ef028dd7ba89e992a7a04492"}
Nov 03 14:24:54.216253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.209516    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e725205961a251b729a648744ec0c2a49b435624ef028dd7ba89e992a7a04492"
Nov 03 14:24:54.216253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.209594    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"
Nov 03 14:24:54.216919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.216752    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" event={"ID":"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5","Type":"ContainerStarted","Data":"49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a"}
Nov 03 14:24:54.216919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.216809    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" event={"ID":"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5","Type":"ContainerStarted","Data":"f1d01790d7b6cf946bed6e0d2cc1af21eaed0b20ec28791ec18926ba0ec75e37"}
Nov 03 14:24:54.216919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.216879    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:24:54.220744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.220711    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" event={"ID":"a8e30eee-5488-49a1-ad18-9b2ca9bd7224","Type":"ContainerStarted","Data":"4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4"}
Nov 03 14:24:54.220842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.220767    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:24:54.220842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.220793    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" event={"ID":"a8e30eee-5488-49a1-ad18-9b2ca9bd7224","Type":"ContainerStarted","Data":"4ecabe8b85b16b0707e26f109440ec62b3593c3183edc2afb527f762e2a94cfc"}
Nov 03 14:24:54.270895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.270846    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" podStartSLOduration=3.270830922 podStartE2EDuration="3.270830922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:54.270477464 +0000 UTC m=+772.320508646" watchObservedRunningTime="2026-11-03 14:24:54.270830922 +0000 UTC m=+772.320862107"
Nov 03 14:24:54.315787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.315741    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" podStartSLOduration=3.315728792 podStartE2EDuration="3.315728792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:54.313092486 +0000 UTC m=+772.363123669" watchObservedRunningTime="2026-11-03 14:24:54.315728792 +0000 UTC m=+772.365759987"
Nov 03 14:24:54.817939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:54.817901    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:56.726219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:56.726187    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:57.117521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:57.117443    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:57.333982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:57.333946    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:57.986031 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:24:57.985988    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:24:58.250301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:58.250197    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:58.896834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:58.896758    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:24:59.002190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.002158    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"]
Nov 03 14:24:59.002778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.002763    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2651061b-aedc-49a6-a2c0-aa645a74a7c6" containerName="revoke-keys"
Nov 03 14:24:59.002824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.002780    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2651061b-aedc-49a6-a2c0-aa645a74a7c6" containerName="revoke-keys"
Nov 03 14:24:59.002876 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.002866    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2651061b-aedc-49a6-a2c0-aa645a74a7c6" containerName="revoke-keys"
Nov 03 14:24:59.006697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.006679    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.029893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.029866    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"]
Nov 03 14:24:59.039589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.039559    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:24:59.039770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.039698    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:24:59.039770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.039750    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:24:59.039884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.039836    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:24:59.095697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.095666    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.095893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.095736    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27wg\" (UniqueName: \"kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.096024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.095999    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.196918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.196887    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.197057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.196979    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.197057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.197043    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27wg\" (UniqueName: \"kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.197603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.197572    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.199374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.199352    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.220163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.220142    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27wg\" (UniqueName: \"kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg\") pod \"maas-api-revoke-sub-af6f73ee-c26q8\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.324502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.324479    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:24:59.644297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:24:59.644247    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"]
Nov 03 14:24:59.644631 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:24:59.644603    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf920a72_cd34_40d7_b34d_defdecf3acf0.slice/crio-4c9b57d232b5bc5073f04391006db7332649c9d558e38c48e2980daa6cc569f4 WatchSource:0}: Error finding container 4c9b57d232b5bc5073f04391006db7332649c9d558e38c48e2980daa6cc569f4: Status 404 returned error can't find the container with id 4c9b57d232b5bc5073f04391006db7332649c9d558e38c48e2980daa6cc569f4
Nov 03 14:25:00.116177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:00.116149    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:25:00.263328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:00.263253    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8" event={"ID":"cf920a72-cd34-40d7-b34d-defdecf3acf0","Type":"ContainerStarted","Data":"3d26fbc6f0147b9c25b62e43c07876ce5d67e4fc23bbf59ff9ea8dc3a4d2bdaa"}
Nov 03 14:25:00.263328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:00.263301    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8" event={"ID":"cf920a72-cd34-40d7-b34d-defdecf3acf0","Type":"ContainerStarted","Data":"4c9b57d232b5bc5073f04391006db7332649c9d558e38c48e2980daa6cc569f4"}
Nov 03 14:25:00.318881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:00.318832    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8" podStartSLOduration=2.3188182 podStartE2EDuration="2.3188182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:00.317397785 +0000 UTC m=+778.367428970" watchObservedRunningTime="2026-11-03 14:25:00.3188182 +0000 UTC m=+778.368849383"
Nov 03 14:25:01.157253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:01.157049    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:01.306372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:01.306336    6481 generic.go:356] "Generic (PLEG): container finished" podID="cf920a72-cd34-40d7-b34d-defdecf3acf0" containerID="3d26fbc6f0147b9c25b62e43c07876ce5d67e4fc23bbf59ff9ea8dc3a4d2bdaa" exitCode=0
Nov 03 14:25:01.306840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:01.306800    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8" event={"ID":"cf920a72-cd34-40d7-b34d-defdecf3acf0","Type":"ContainerDied","Data":"3d26fbc6f0147b9c25b62e43c07876ce5d67e4fc23bbf59ff9ea8dc3a4d2bdaa"}
Nov 03 14:25:02.340650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:02.340605    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:03.509437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.509089    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:25:03.516947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.516920    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:03.528669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.526222    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg\" (UniqueName: \"kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg\") pod \"cf920a72-cd34-40d7-b34d-defdecf3acf0\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") "
Nov 03 14:25:03.528669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.528326    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca\") pod \"cf920a72-cd34-40d7-b34d-defdecf3acf0\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") "
Nov 03 14:25:03.528669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.528484    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca\") pod \"cf920a72-cd34-40d7-b34d-defdecf3acf0\" (UID: \"cf920a72-cd34-40d7-b34d-defdecf3acf0\") "
Nov 03 14:25:03.533591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.529367    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca" pod "cf920a72-cd34-40d7-b34d-defdecf3acf0" (UID: "cf920a72-cd34-40d7-b34d-defdecf3acf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:25:03.533591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.533039    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca" pod "cf920a72-cd34-40d7-b34d-defdecf3acf0" (UID: "cf920a72-cd34-40d7-b34d-defdecf3acf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:03.534431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.534286    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg" pod "cf920a72-cd34-40d7-b34d-defdecf3acf0" (UID: "cf920a72-cd34-40d7-b34d-defdecf3acf0"). InnerVolumeSpecName "kube-api-access-h27wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:03.601840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.601793    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8" event={"ID":"cf920a72-cd34-40d7-b34d-defdecf3acf0","Type":"ContainerDied","Data":"4c9b57d232b5bc5073f04391006db7332649c9d558e38c48e2980daa6cc569f4"}
Nov 03 14:25:03.602019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.601874    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c9b57d232b5bc5073f04391006db7332649c9d558e38c48e2980daa6cc569f4"
Nov 03 14:25:03.602019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.601927    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"
Nov 03 14:25:03.630290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.630265    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:03.630290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.630289    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h27wg\" (UniqueName: \"kubernetes.io/projected/cf920a72-cd34-40d7-b34d-defdecf3acf0-kube-api-access-h27wg\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:03.630447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:03.630301    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf920a72-cd34-40d7-b34d-defdecf3acf0-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:05.229582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:05.229549    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:25:05.230679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:05.230660    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:25:05.468626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:05.468587    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:05.699606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:05.699557    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:07.733390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:07.733357    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:09.535092 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.535057    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"]
Nov 03 14:25:09.535863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.535843    6481 cpu_manager.go:414] "containerMap: removing container" podUID="cf920a72-cd34-40d7-b34d-defdecf3acf0" containerName="revoke-keys"
Nov 03 14:25:09.535976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.535863    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf920a72-cd34-40d7-b34d-defdecf3acf0" containerName="revoke-keys"
Nov 03 14:25:09.536080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.536058    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf920a72-cd34-40d7-b34d-defdecf3acf0" containerName="revoke-keys"
Nov 03 14:25:09.544138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.544112    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.560749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.560724    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"]
Nov 03 14:25:09.577186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.577159    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:25:09.577402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.577185    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:25:09.577402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.577309    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:25:09.577402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.577352    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:25:09.611764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.611738    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.611918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.611782    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.611974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.611963    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fqzf\" (UniqueName: \"kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.716098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.716061    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fqzf\" (UniqueName: \"kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.716314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.716198    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.716314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.716251    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.717115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.717082    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.726374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.726317    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.738780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.738757    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fqzf\" (UniqueName: \"kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf\") pod \"maas-api-revoke-sub-16a15a83-ftb67\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.865528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.865443    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:09.955271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:09.954465    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:10.254893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.254863    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"]
Nov 03 14:25:10.258939 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:10.258900    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623ce384_d214_420d_aa2e_25d3b4955e24.slice/crio-79a4d5e043d776051372111c0aaf144b3b409eba9b609283941f3324fe378596 WatchSource:0}: Error finding container 79a4d5e043d776051372111c0aaf144b3b409eba9b609283941f3324fe378596: Status 404 returned error can't find the container with id 79a4d5e043d776051372111c0aaf144b3b409eba9b609283941f3324fe378596
Nov 03 14:25:10.483630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.483588    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"]
Nov 03 14:25:10.521301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.492159    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552032    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552117    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2c8\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552180    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552268    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552304    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552337    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552373    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552409    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.552793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.552638    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.580374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.580343    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"]
Nov 03 14:25:10.654276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.653960    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.654276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.654013    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.654276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.654082    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2c8\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.654276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.654137    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.655057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655026    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.655170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655070    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.655170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655100    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.655170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655131    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.655170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655164    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.656370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655605    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.656370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655712    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.656370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655893    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.656370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.655973    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.656684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.656561    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.659268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.659165    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.660780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.660756    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.678816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.678792    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2c8\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.679430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.679387    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:10.693249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.687512    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67" event={"ID":"623ce384-d214-420d-aa2e-25d3b4955e24","Type":"ContainerStarted","Data":"79a4d5e043d776051372111c0aaf144b3b409eba9b609283941f3324fe378596"}
Nov 03 14:25:10.834441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:10.834377    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:11.460255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.437742    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"]
Nov 03 14:25:11.527544 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:11.527497    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f39e7c_6b8b_44d4_a160_3c921c2288ac.slice/crio-1f12dc0a8c177945b865f5a32b0aacc43f63e313371c2a6119f7de1f5c303a17 WatchSource:0}: Error finding container 1f12dc0a8c177945b865f5a32b0aacc43f63e313371c2a6119f7de1f5c303a17: Status 404 returned error can't find the container with id 1f12dc0a8c177945b865f5a32b0aacc43f63e313371c2a6119f7de1f5c303a17
Nov 03 14:25:11.591623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.589963    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:11.591623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.590059    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:11.591623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.590108    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:11.741270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.740623    6481 generic.go:356] "Generic (PLEG): container finished" podID="623ce384-d214-420d-aa2e-25d3b4955e24" containerID="2fd9e9eaeca62ee1027426e5c2f23307a22617929d9971b2264d49adf7e11d88" exitCode=0
Nov 03 14:25:11.741270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.740830    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67" event={"ID":"623ce384-d214-420d-aa2e-25d3b4955e24","Type":"ContainerDied","Data":"2fd9e9eaeca62ee1027426e5c2f23307a22617929d9971b2264d49adf7e11d88"}
Nov 03 14:25:11.755308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:11.755277    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" event={"ID":"53f39e7c-6b8b-44d4-a160-3c921c2288ac","Type":"ContainerStarted","Data":"1f12dc0a8c177945b865f5a32b0aacc43f63e313371c2a6119f7de1f5c303a17"}
Nov 03 14:25:12.151396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.150713    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:12.319372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.319339    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:12.771314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.771279    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" event={"ID":"53f39e7c-6b8b-44d4-a160-3c921c2288ac","Type":"ContainerStarted","Data":"d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2"}
Nov 03 14:25:12.834635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.834605    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:12.839370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.839335    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:12.839503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.839391    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:12.839541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:12.839484    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podStartSLOduration=2.839469759 podStartE2EDuration="2.839469759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:12.8349551 +0000 UTC m=+790.884986284" watchObservedRunningTime="2026-11-03 14:25:12.839469759 +0000 UTC m=+790.889500943"
Nov 03 14:25:13.036463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.036375    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:13.231190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.231157    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf\" (UniqueName: \"kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf\") pod \"623ce384-d214-420d-aa2e-25d3b4955e24\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") "
Nov 03 14:25:13.231362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.231250    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca\") pod \"623ce384-d214-420d-aa2e-25d3b4955e24\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") "
Nov 03 14:25:13.231429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.231395    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca\") pod \"623ce384-d214-420d-aa2e-25d3b4955e24\" (UID: \"623ce384-d214-420d-aa2e-25d3b4955e24\") "
Nov 03 14:25:13.231679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.231651    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca" pod "623ce384-d214-420d-aa2e-25d3b4955e24" (UID: "623ce384-d214-420d-aa2e-25d3b4955e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:25:13.231964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.231945    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:13.233581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.233554    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca" pod "623ce384-d214-420d-aa2e-25d3b4955e24" (UID: "623ce384-d214-420d-aa2e-25d3b4955e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:13.234496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.234474    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf" pod "623ce384-d214-420d-aa2e-25d3b4955e24" (UID: "623ce384-d214-420d-aa2e-25d3b4955e24"). InnerVolumeSpecName "kube-api-access-9fqzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:13.337750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.337309    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/623ce384-d214-420d-aa2e-25d3b4955e24-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:13.337750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.337349    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fqzf\" (UniqueName: \"kubernetes.io/projected/623ce384-d214-420d-aa2e-25d3b4955e24-kube-api-access-9fqzf\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:13.387060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.386973    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"]
Nov 03 14:25:13.387757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.387729    6481 cpu_manager.go:414] "containerMap: removing container" podUID="623ce384-d214-420d-aa2e-25d3b4955e24" containerName="revoke-keys"
Nov 03 14:25:13.387923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.387759    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623ce384-d214-420d-aa2e-25d3b4955e24" containerName="revoke-keys"
Nov 03 14:25:13.387923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.387901    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="623ce384-d214-420d-aa2e-25d3b4955e24" containerName="revoke-keys"
Nov 03 14:25:13.391672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.391648    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.434600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.434563    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"]
Nov 03 14:25:13.539691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.539657    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.539851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.539706    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlqt\" (UniqueName: \"kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.539851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.539778    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.640755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.640668    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.640755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.640718    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlqt\" (UniqueName: \"kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.640983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.640755    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.641388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.641366    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.643141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.643118    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.663061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.663040    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlqt\" (UniqueName: \"kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt\") pod \"maas-api-revoke-sub-ab3b030e-jjrcd\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.737406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.737370    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:13.778173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.778148    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"
Nov 03 14:25:13.778930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.778900    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67" event={"ID":"623ce384-d214-420d-aa2e-25d3b4955e24","Type":"ContainerDied","Data":"79a4d5e043d776051372111c0aaf144b3b409eba9b609283941f3324fe378596"}
Nov 03 14:25:13.779037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.778952    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79a4d5e043d776051372111c0aaf144b3b409eba9b609283941f3324fe378596"
Nov 03 14:25:13.835768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.835729    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:13.835928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:13.835793    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:14.039099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.039071    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"]
Nov 03 14:25:14.039846 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:14.039821    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bf7d5e_cde5_4ee0_8d27_5597ba64fc09.slice/crio-fddfe42e47b33a6501624ab33c585099292906c84eab4e8b86ff9184e82e05a3 WatchSource:0}: Error finding container fddfe42e47b33a6501624ab33c585099292906c84eab4e8b86ff9184e82e05a3: Status 404 returned error can't find the container with id fddfe42e47b33a6501624ab33c585099292906c84eab4e8b86ff9184e82e05a3
Nov 03 14:25:14.384052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.383968    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:14.786187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.786147    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd" event={"ID":"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09","Type":"ContainerStarted","Data":"40fdb28ec759a6817bff5c9c9770002b3888c348fac73913f6bf41a421a0eece"}
Nov 03 14:25:14.786674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.786200    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd" event={"ID":"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09","Type":"ContainerStarted","Data":"fddfe42e47b33a6501624ab33c585099292906c84eab4e8b86ff9184e82e05a3"}
Nov 03 14:25:14.835672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.835637    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:14.835857 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.835706    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:14.838617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:14.838564    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd" podStartSLOduration=1.8385514280000002 podStartE2EDuration="1.838551428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:14.836055638 +0000 UTC m=+792.886086820" watchObservedRunningTime="2026-11-03 14:25:14.838551428 +0000 UTC m=+792.888582611"
Nov 03 14:25:15.576472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.576441    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"]
Nov 03 14:25:15.580411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.580390    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.633735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.633701    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5b97f1-openshift-default-dockercfg-gbzmt\""
Nov 03 14:25:15.636940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.636901    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"]
Nov 03 14:25:15.667714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.667683    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.667851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.667742    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.667851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.667762    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.667851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.667809    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.667969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.667935    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.668012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.667981    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.668097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.668080    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x8n\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.668152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.668136    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.668248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.668190    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.769824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.769784    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.769995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.769834    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.769995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.769876    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x8n\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.769995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.769911    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770025    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770172    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770214    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770327    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770385    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770416    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770721    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770783    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770801    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.770864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.770851    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.772756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.772733    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.772891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.772874    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.791151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.791126    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.791467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.791165    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x8n\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:15.794847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.794821    6481 generic.go:356] "Generic (PLEG): container finished" podID="09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" containerID="40fdb28ec759a6817bff5c9c9770002b3888c348fac73913f6bf41a421a0eece" exitCode=0
Nov 03 14:25:15.794931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.794895    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd" event={"ID":"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09","Type":"ContainerDied","Data":"40fdb28ec759a6817bff5c9c9770002b3888c348fac73913f6bf41a421a0eece"}
Nov 03 14:25:15.834891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.834813    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:15.834891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.834874    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:15.895468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:15.895437    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:16.439604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.439560    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:16.517274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.499701    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"]
Nov 03 14:25:16.531258 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:16.529753    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d2777d_f0f8_4f2e_811e_47151be12b1d.slice/crio-8b408f0248015ea44a3bbe63741d291fff6d3bfbb666342d83cfd05c9fcdea3e WatchSource:0}: Error finding container 8b408f0248015ea44a3bbe63741d291fff6d3bfbb666342d83cfd05c9fcdea3e: Status 404 returned error can't find the container with id 8b408f0248015ea44a3bbe63741d291fff6d3bfbb666342d83cfd05c9fcdea3e
Nov 03 14:25:16.532593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.532559    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:16.532698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.532631    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:16.532698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.532663    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:16.829488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.829297    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" event={"ID":"d8d2777d-f0f8-4f2e-811e-47151be12b1d","Type":"ContainerStarted","Data":"be29bf6fb765d20445fea83da87ae2581f19b6d2641a9cada1d607967bb7aaa4"}
Nov 03 14:25:16.829488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.829361    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" event={"ID":"d8d2777d-f0f8-4f2e-811e-47151be12b1d","Type":"ContainerStarted","Data":"8b408f0248015ea44a3bbe63741d291fff6d3bfbb666342d83cfd05c9fcdea3e"}
Nov 03 14:25:16.835186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.835162    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:16.835300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.835207    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:16.883436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.883370    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podStartSLOduration=1.883352264 podStartE2EDuration="1.883352264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:16.878885766 +0000 UTC m=+794.928916949" watchObservedRunningTime="2026-11-03 14:25:16.883352264 +0000 UTC m=+794.933383448"
Nov 03 14:25:16.899890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:16.896459    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:17.031451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.031418    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:17.147330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.144038    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca\") pod \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") "
Nov 03 14:25:17.147330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.144105    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca\") pod \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") "
Nov 03 14:25:17.147330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.144188    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt\" (UniqueName: \"kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt\") pod \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\" (UID: \"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09\") "
Nov 03 14:25:17.147330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.146798    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca" pod "09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" (UID: "09bf7d5e-cde5-4ee0-8d27-5597ba64fc09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:25:17.167255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.154458    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca" pod "09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" (UID: "09bf7d5e-cde5-4ee0-8d27-5597ba64fc09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:17.167255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.154574    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt" pod "09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" (UID: "09bf7d5e-cde5-4ee0-8d27-5597ba64fc09"). InnerVolumeSpecName "kube-api-access-bzlqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:17.254478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.254441    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:17.254782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.254761    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:17.254902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.254887    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlqt\" (UniqueName: \"kubernetes.io/projected/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09-kube-api-access-bzlqt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:17.840251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.839682    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:17.840251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.839747    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:17.856727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.856696    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"
Nov 03 14:25:17.856727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.856707    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd" event={"ID":"09bf7d5e-cde5-4ee0-8d27-5597ba64fc09","Type":"ContainerDied","Data":"fddfe42e47b33a6501624ab33c585099292906c84eab4e8b86ff9184e82e05a3"}
Nov 03 14:25:17.856945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.856782    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fddfe42e47b33a6501624ab33c585099292906c84eab4e8b86ff9184e82e05a3"
Nov 03 14:25:17.897430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.897396    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 14:25:17.897562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:17.897447    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 14:25:18.518254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:18.517600    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:18.835823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:18.835734    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Nov 03 14:25:18.835823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:18.835800    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Nov 03 14:25:18.896215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:18.896177    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 14:25:18.896615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:18.896251    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 14:25:19.839578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:19.839543    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:19.839920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:19.839895    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:19.840698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:19.840677    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:25:19.896724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:19.896694    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 14:25:19.897130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:19.896810    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 14:25:20.632920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:20.632879    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:20.896611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:20.896528    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 14:25:20.896611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:20.896583    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 14:25:21.524545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:21.524497    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:25:21.777790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:21.777711    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"]
Nov 03 14:25:21.900289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:21.898104    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 14:25:21.900289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:21.898258    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 14:25:21.983338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:21.983068    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:22.890254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:22.882153    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:22.913265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:22.899880    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Nov 03 14:25:22.913265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:22.899973    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Nov 03 14:25:23.606847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:23.605817    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:25:24.137252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.136414    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:24.280256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.277703    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:24.314251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.306508    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:24.501253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.485363    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:24.501253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.490304    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:24.509252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.506829    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:24.613549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.611302    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:25:24.613549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.612033    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:25:24.829317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.826281    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:24.829861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.829839    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:24.872449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:24.872414    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:25:25.217688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.215942    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 14:25:25.217688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.216643    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:25:25.316350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.312576    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:25:25.338208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.326178    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:25:25.512418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.509745    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:25.512418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.510175    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:25:25.512418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.510996    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:25.512418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.511281    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:25:25.512418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.511576    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:25:25.512418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.511648    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:25:25.626463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.626420    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:25.665331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.665297    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"]
Nov 03 14:25:25.685261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.666382    6481 cpu_manager.go:414] "containerMap: removing container" podUID="09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" containerName="revoke-keys"
Nov 03 14:25:25.685500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.685475    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" containerName="revoke-keys"
Nov 03 14:25:25.696275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:25.685842    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" containerName="revoke-keys"
Nov 03 14:25:26.240297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.223307    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:26.266511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.253289    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.299317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.286015    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e759e5-openshift-default-dockercfg-ldb9g\""
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.325706    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.326793    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.326874    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hjbx\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.326919    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.326949    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.327032    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.327075    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.327096    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.327117    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.352249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.327177    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.394119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.394073    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:26.415264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.406110    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"]
Nov 03 14:25:26.415264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.406220    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:25:26.415264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.406466    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:25:26.415264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.406754    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:25:26.415264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.406803    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:25:26.415264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.406836    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:25:26.429534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.429503    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.430163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.430072    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.430678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.430656    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.430817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.430662    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.430980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.430961    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hjbx\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.431591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.431570    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.431743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.431727    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.431920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.431906    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.432343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.432325    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.432466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.432452    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.432593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.432579    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.433211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.433190    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.434284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.433915    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.435533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.435512    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.436506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.436473    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.455154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.455129    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:25:26.474910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.459702    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.478333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.475700    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.478333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.478304    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hjbx\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx\") pod \"e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") " pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:26.720609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:26.720577    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:27.079875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.079847    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"]
Nov 03 14:25:27.080755 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:27.080726    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19dd7a32_708d_43ca_97a3_29f8d9d77578.slice/crio-f30d9857aee6f0d0ae3818cb73be27852e3285920a00921b27ac8a93486456de WatchSource:0}: Error finding container f30d9857aee6f0d0ae3818cb73be27852e3285920a00921b27ac8a93486456de: Status 404 returned error can't find the container with id f30d9857aee6f0d0ae3818cb73be27852e3285920a00921b27ac8a93486456de
Nov 03 14:25:27.083311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.083282    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:27.083390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.083340    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:27.083390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.083371    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:25:27.135357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.135328    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"]
Nov 03 14:25:27.289255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.288819    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:27.328438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.328347    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" event={"ID":"19dd7a32-708d-43ca-97a3-29f8d9d77578","Type":"ContainerStarted","Data":"56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989"}
Nov 03 14:25:27.328438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.328409    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" event={"ID":"19dd7a32-708d-43ca-97a3-29f8d9d77578","Type":"ContainerStarted","Data":"f30d9857aee6f0d0ae3818cb73be27852e3285920a00921b27ac8a93486456de"}
Nov 03 14:25:27.422135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.422082    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podStartSLOduration=2.4220685 podStartE2EDuration="2.4220685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:27.420950268 +0000 UTC m=+805.470981452" watchObservedRunningTime="2026-11-03 14:25:27.4220685 +0000 UTC m=+805.472099682"
Nov 03 14:25:27.725357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:27.725317    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:27.986363 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:25:27.986284    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:25:28.728364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:28.727934    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:28.728364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:28.727991    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:28.825812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:28.825768    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:29.725468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:29.725430    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:29.725694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:29.725492    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:29.808729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:29.808696    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:30.626558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.626517    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"]
Nov 03 14:25:30.631972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.631953    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.638576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.638547    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"]
Nov 03 14:25:30.684691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.684653    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:25:30.684691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.684672    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:25:30.684960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.684740    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:25:30.684960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.684788    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:25:30.730431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.729640    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:30.730431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.729696    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:30.730728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.730620    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.730728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.730703    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.730877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.730855    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq2gr\" (UniqueName: \"kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.731164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.731137    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:30.832405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.832367    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq2gr\" (UniqueName: \"kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.832789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.832419    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.832789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.832530    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.833333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.833310    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.835042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.835020    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.858826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.858795    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq2gr\" (UniqueName: \"kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr\") pod \"maas-api-revoke-sub-55746ebb-j6566\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:30.978401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:30.978366    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:31.227884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:31.227060    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:31.557917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:31.553888    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"]
Nov 03 14:25:31.732464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:31.728027    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:31.732464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:31.728108    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:31.752283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:31.736090    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:32.007308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:32.000512    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:32.504936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:32.504780    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566" event={"ID":"92e93b4f-b94b-4683-bb07-d5063fe45c06","Type":"ContainerStarted","Data":"e9eece00f2953322a55abfda362c1f57339d2bb6e4c0eed82b055bd799019465"}
Nov 03 14:25:32.731272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:32.729417    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:32.731272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:32.729492    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:33.321255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.317339    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:25:33.356253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.340433    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:25:33.408255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.407590    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:33.471555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.471457    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:33.496353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.496307    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded"
Nov 03 14:25:33.505392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.505353    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:33.602189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.572582    6481 generic.go:356] "Generic (PLEG): container finished" podID="92e93b4f-b94b-4683-bb07-d5063fe45c06" containerID="0eb2f4fff5340baf6b5eb3bdca98f25436e530ce58f8cb335662f9af3a0da299" exitCode=0
Nov 03 14:25:33.602189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.572759    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566" event={"ID":"92e93b4f-b94b-4683-bb07-d5063fe45c06","Type":"ContainerDied","Data":"0eb2f4fff5340baf6b5eb3bdca98f25436e530ce58f8cb335662f9af3a0da299"}
Nov 03 14:25:33.603596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.603511    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:33.722878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.722762    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:33.723177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:33.723126    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:34.285405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.285357    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:34.318354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.318306    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:34.318988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.318963    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:25:34.367014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.366972    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:25:34.723268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.722617    6481 patch_prober.go:29] interesting pod/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Nov 03 14:25:34.723268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.722714    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Nov 03 14:25:34.782176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.782136    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:34.794973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.794943    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr\" (UniqueName: \"kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr\") pod \"92e93b4f-b94b-4683-bb07-d5063fe45c06\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") "
Nov 03 14:25:34.795122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.795098    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca\") pod \"92e93b4f-b94b-4683-bb07-d5063fe45c06\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") "
Nov 03 14:25:34.795462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.795437    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca" pod "92e93b4f-b94b-4683-bb07-d5063fe45c06" (UID: "92e93b4f-b94b-4683-bb07-d5063fe45c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:25:34.795659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.795644    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:34.797344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.797316    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr" pod "92e93b4f-b94b-4683-bb07-d5063fe45c06" (UID: "92e93b4f-b94b-4683-bb07-d5063fe45c06"). InnerVolumeSpecName "kube-api-access-fq2gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:34.848793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.848714    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"]
Nov 03 14:25:34.849842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.849681    6481 cpu_manager.go:414] "containerMap: removing container" podUID="92e93b4f-b94b-4683-bb07-d5063fe45c06" containerName="revoke-keys"
Nov 03 14:25:34.849842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.849706    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e93b4f-b94b-4683-bb07-d5063fe45c06" containerName="revoke-keys"
Nov 03 14:25:34.851264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.849949    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e93b4f-b94b-4683-bb07-d5063fe45c06" containerName="revoke-keys"
Nov 03 14:25:34.854686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.854670    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:34.874186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.874153    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"]
Nov 03 14:25:34.896166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.896138    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca\") pod \"92e93b4f-b94b-4683-bb07-d5063fe45c06\" (UID: \"92e93b4f-b94b-4683-bb07-d5063fe45c06\") "
Nov 03 14:25:34.896931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.896908    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq2gr\" (UniqueName: \"kubernetes.io/projected/92e93b4f-b94b-4683-bb07-d5063fe45c06-kube-api-access-fq2gr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:34.899849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.899820    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca" pod "92e93b4f-b94b-4683-bb07-d5063fe45c06" (UID: "92e93b4f-b94b-4683-bb07-d5063fe45c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:34.998456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.998420    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s8l\" (UniqueName: \"kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:34.998631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.998463    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:34.998765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.998741    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:34.998839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:34.998804    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92e93b4f-b94b-4683-bb07-d5063fe45c06-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:35.099916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.099819    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58s8l\" (UniqueName: \"kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.099916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.099876    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.100158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.100142    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.100606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.100581    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.102884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.102833    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.122528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.122505    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s8l\" (UniqueName: \"kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l\") pod \"maas-api-revoke-sub-a92ede12-jwjzc\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.175143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.175122    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:35.479902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.479864    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"]
Nov 03 14:25:35.480211 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:35.480007    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee5a510_faca_4805_b851_48030c4ff04f.slice/crio-baebc13a53dcef73b466f299edfa5be88b0b374b264d4db1834f2fd9e121c69b WatchSource:0}: Error finding container baebc13a53dcef73b466f299edfa5be88b0b374b264d4db1834f2fd9e121c69b: Status 404 returned error can't find the container with id baebc13a53dcef73b466f299edfa5be88b0b374b264d4db1834f2fd9e121c69b
Nov 03 14:25:35.588825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.588795    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"
Nov 03 14:25:35.588975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.588828    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566" event={"ID":"92e93b4f-b94b-4683-bb07-d5063fe45c06","Type":"ContainerDied","Data":"e9eece00f2953322a55abfda362c1f57339d2bb6e4c0eed82b055bd799019465"}
Nov 03 14:25:35.588975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.588905    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9eece00f2953322a55abfda362c1f57339d2bb6e4c0eed82b055bd799019465"
Nov 03 14:25:35.590203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.590170    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc" event={"ID":"8ee5a510-faca-4805-b851-48030c4ff04f","Type":"ContainerStarted","Data":"baebc13a53dcef73b466f299edfa5be88b0b374b264d4db1834f2fd9e121c69b"}
Nov 03 14:25:35.704446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.704412    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:35.726570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.726542    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:35.727067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.727021    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:35.727877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:35.727859    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:25:36.607193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:36.607164    6481 generic.go:356] "Generic (PLEG): container finished" podID="8ee5a510-faca-4805-b851-48030c4ff04f" containerID="8760aba4c7ba47d354038b3844f2cf6be0123f20936963c2bc04db52eba8c917" exitCode=0
Nov 03 14:25:36.607547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:36.607261    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc" event={"ID":"8ee5a510-faca-4805-b851-48030c4ff04f","Type":"ContainerDied","Data":"8760aba4c7ba47d354038b3844f2cf6be0123f20936963c2bc04db52eba8c917"}
Nov 03 14:25:36.707323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:36.707246    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:36.798371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:36.798340    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main" containerID="cri-o://3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65" gracePeriod=45
Nov 03 14:25:37.058967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.058937    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:37.068393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.068376    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:25:37.123836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.123808    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home\") pod \"dc109d21-9632-48ec-8f57-5e143137f3e9\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") "
Nov 03 14:25:37.123960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.123848    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache\") pod \"dc109d21-9632-48ec-8f57-5e143137f3e9\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") "
Nov 03 14:25:37.124029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.123986    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb\" (UniqueName: \"kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb\") pod \"dc109d21-9632-48ec-8f57-5e143137f3e9\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") "
Nov 03 14:25:37.125253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.124092    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home" pod "dc109d21-9632-48ec-8f57-5e143137f3e9" (UID: "dc109d21-9632-48ec-8f57-5e143137f3e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:37.125253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.124109    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm\") pod \"dc109d21-9632-48ec-8f57-5e143137f3e9\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") "
Nov 03 14:25:37.125253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.124174    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs\") pod \"dc109d21-9632-48ec-8f57-5e143137f3e9\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") "
Nov 03 14:25:37.125253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.124303    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir\") pod \"dc109d21-9632-48ec-8f57-5e143137f3e9\" (UID: \"dc109d21-9632-48ec-8f57-5e143137f3e9\") "
Nov 03 14:25:37.125253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.124400    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache" pod "dc109d21-9632-48ec-8f57-5e143137f3e9" (UID: "dc109d21-9632-48ec-8f57-5e143137f3e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:37.125253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.124698    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir" pod "dc109d21-9632-48ec-8f57-5e143137f3e9" (UID: "dc109d21-9632-48ec-8f57-5e143137f3e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:37.125710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.125625    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.125710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.125648    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.125710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.125662    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.127338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.127314    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb" pod "dc109d21-9632-48ec-8f57-5e143137f3e9" (UID: "dc109d21-9632-48ec-8f57-5e143137f3e9"). InnerVolumeSpecName "kube-api-access-h76sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:37.128749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.128482    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs" pod "dc109d21-9632-48ec-8f57-5e143137f3e9" (UID: "dc109d21-9632-48ec-8f57-5e143137f3e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:37.130339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.130315    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm" pod "dc109d21-9632-48ec-8f57-5e143137f3e9" (UID: "dc109d21-9632-48ec-8f57-5e143137f3e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:37.226848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.226768    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h76sb\" (UniqueName: \"kubernetes.io/projected/dc109d21-9632-48ec-8f57-5e143137f3e9-kube-api-access-h76sb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.226848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.226805    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc109d21-9632-48ec-8f57-5e143137f3e9-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.226848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.226820    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc109d21-9632-48ec-8f57-5e143137f3e9-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.286361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.286320    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:37.615050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.614968    6481 generic.go:356] "Generic (PLEG): container finished" podID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerID="3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65" exitCode=0
Nov 03 14:25:37.615050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.615011    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"
Nov 03 14:25:37.615524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.615048    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" event={"ID":"dc109d21-9632-48ec-8f57-5e143137f3e9","Type":"ContainerDied","Data":"3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65"}
Nov 03 14:25:37.615524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.615085    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c" event={"ID":"dc109d21-9632-48ec-8f57-5e143137f3e9","Type":"ContainerDied","Data":"b390290ae45a9cbd1413752d2a64ce98d1f82e0060304d9f119c2b9194ee37ab"}
Nov 03 14:25:37.615524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.615105    6481 scope.go:122] "RemoveContainer" containerID="3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65"
Nov 03 14:25:37.631920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.631897    6481 scope.go:122] "RemoveContainer" containerID="3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65"
Nov 03 14:25:37.632271 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:25:37.632220    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65\": container with ID starting with 3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65 not found: ID does not exist" containerID="3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65"
Nov 03 14:25:37.632375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.632283    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65"} err="failed to get container status \"3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65\": rpc error: code = NotFound desc = could not find container \"3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65\": container with ID starting with 3c8309e56f9f438f446a2768e902322c5c1705616c38dbab800ce24c33a1df65 not found: ID does not exist"
Nov 03 14:25:37.673058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.673030    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"]
Nov 03 14:25:37.685357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.685329    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-1ffaa0/auth-test-model-1ffaa0-kserve-76899c5d49-fnc9c"]
Nov 03 14:25:37.761179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.761154    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:37.833677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.833646    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca\") pod \"8ee5a510-faca-4805-b851-48030c4ff04f\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") "
Nov 03 14:25:37.833825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.833806    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca\") pod \"8ee5a510-faca-4805-b851-48030c4ff04f\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") "
Nov 03 14:25:37.833939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.833926    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l\" (UniqueName: \"kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l\") pod \"8ee5a510-faca-4805-b851-48030c4ff04f\" (UID: \"8ee5a510-faca-4805-b851-48030c4ff04f\") "
Nov 03 14:25:37.834091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.834065    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca" pod "8ee5a510-faca-4805-b851-48030c4ff04f" (UID: "8ee5a510-faca-4805-b851-48030c4ff04f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:25:37.834326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.834308    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.835967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.835945    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca" pod "8ee5a510-faca-4805-b851-48030c4ff04f" (UID: "8ee5a510-faca-4805-b851-48030c4ff04f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:37.836475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.836458    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l" pod "8ee5a510-faca-4805-b851-48030c4ff04f" (UID: "8ee5a510-faca-4805-b851-48030c4ff04f"). InnerVolumeSpecName "kube-api-access-58s8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:37.935699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.935672    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee5a510-faca-4805-b851-48030c4ff04f-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:37.935699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:37.935697    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s8l\" (UniqueName: \"kubernetes.io/projected/8ee5a510-faca-4805-b851-48030c4ff04f-kube-api-access-58s8l\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:38.067745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:38.067712    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" path="/var/lib/kubelet/pods/dc109d21-9632-48ec-8f57-5e143137f3e9/volumes"
Nov 03 14:25:38.160752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:38.160726    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:38.623407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:38.623378    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"
Nov 03 14:25:38.623853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:38.623410    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc" event={"ID":"8ee5a510-faca-4805-b851-48030c4ff04f","Type":"ContainerDied","Data":"baebc13a53dcef73b466f299edfa5be88b0b374b264d4db1834f2fd9e121c69b"}
Nov 03 14:25:38.623853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:38.623462    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baebc13a53dcef73b466f299edfa5be88b0b374b264d4db1834f2fd9e121c69b"
Nov 03 14:25:38.968720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:38.968694    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:39.054783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.054754    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"]
Nov 03 14:25:39.055323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.055309    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8ee5a510-faca-4805-b851-48030c4ff04f" containerName="revoke-keys"
Nov 03 14:25:39.055370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.055325    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee5a510-faca-4805-b851-48030c4ff04f" containerName="revoke-keys"
Nov 03 14:25:39.055408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.055382    6481 cpu_manager.go:414] "containerMap: removing container" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main"
Nov 03 14:25:39.055408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.055388    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main"
Nov 03 14:25:39.055471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.055466    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee5a510-faca-4805-b851-48030c4ff04f" containerName="revoke-keys"
Nov 03 14:25:39.055503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.055482    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc109d21-9632-48ec-8f57-5e143137f3e9" containerName="main"
Nov 03 14:25:39.059668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.059651    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.077274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.077254    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b4361f-dockercfg-zmd4n\""
Nov 03 14:25:39.077438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.077420    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b4361f\""
Nov 03 14:25:39.082011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.081988    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"]
Nov 03 14:25:39.126956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.126924    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"]
Nov 03 14:25:39.132314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.132288    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.148888 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.148863    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlb4n\" (UniqueName: \"kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n\") pod \"payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.149045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.148900    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.149045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.148943    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.149045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.149005    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz6mh\" (UniqueName: \"kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh\") pod \"payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.154316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.154291    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"]
Nov 03 14:25:39.231303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.215810    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:39.256983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.256928    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlb4n\" (UniqueName: \"kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n\") pod \"payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.257271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.257248    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.257414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.257400    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.257535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.257519    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz6mh\" (UniqueName: \"kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh\") pod \"payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.259051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.259007    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.260027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.259989    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.308296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.297321    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlb4n\" (UniqueName: \"kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n\") pod \"payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.308296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.297733    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz6mh\" (UniqueName: \"kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh\") pod \"payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.372941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.372909    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:39.450850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.450824    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:39.683000 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:39.682974    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646ad667_a372_4de9_b38e_83332f77a8f5.slice/crio-ea8a18616294ee96b09b2c4510405c1d241678cbe2d9ada4c88c0a110c2272ee WatchSource:0}: Error finding container ea8a18616294ee96b09b2c4510405c1d241678cbe2d9ada4c88c0a110c2272ee: Status 404 returned error can't find the container with id ea8a18616294ee96b09b2c4510405c1d241678cbe2d9ada4c88c0a110c2272ee
Nov 03 14:25:39.683344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.683089    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"]
Nov 03 14:25:39.765677 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:39.765624    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f5d0f1_cab0_496c_b767_68b59f7ae06c.slice/crio-8694b4bb049f480110eec704b2303922942cf3c8ea06e5d5adbc85788ced7748 WatchSource:0}: Error finding container 8694b4bb049f480110eec704b2303922942cf3c8ea06e5d5adbc85788ced7748: Status 404 returned error can't find the container with id 8694b4bb049f480110eec704b2303922942cf3c8ea06e5d5adbc85788ced7748
Nov 03 14:25:39.766625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:39.766602    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"]
Nov 03 14:25:40.449290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.449260    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:40.639420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.639385    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" event={"ID":"646ad667-a372-4de9-b38e-83332f77a8f5","Type":"ContainerStarted","Data":"e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3"}
Nov 03 14:25:40.639603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.639441    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" event={"ID":"646ad667-a372-4de9-b38e-83332f77a8f5","Type":"ContainerStarted","Data":"ea8a18616294ee96b09b2c4510405c1d241678cbe2d9ada4c88c0a110c2272ee"}
Nov 03 14:25:40.639603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.639471    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:40.641126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.641102    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" event={"ID":"88f5d0f1-cab0-496c-b767-68b59f7ae06c","Type":"ContainerStarted","Data":"f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7"}
Nov 03 14:25:40.641289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.641148    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" event={"ID":"88f5d0f1-cab0-496c-b767-68b59f7ae06c","Type":"ContainerStarted","Data":"8694b4bb049f480110eec704b2303922942cf3c8ea06e5d5adbc85788ced7748"}
Nov 03 14:25:40.641289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.641179    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:40.686596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.686548    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" podStartSLOduration=1.686535588 podStartE2EDuration="1.686535588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:40.684587288 +0000 UTC m=+818.734618468" watchObservedRunningTime="2026-11-03 14:25:40.686535588 +0000 UTC m=+818.736566825"
Nov 03 14:25:40.732737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:40.732649    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" podStartSLOduration=1.732634923 podStartE2EDuration="1.732634923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:40.731894314 +0000 UTC m=+818.781925494" watchObservedRunningTime="2026-11-03 14:25:40.732634923 +0000 UTC m=+818.782666106"
Nov 03 14:25:42.058678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.058640    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:25:42.351252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.351140    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" containerID="cri-o://f34bde8e56935a6554b7148c33a6bf31b8c90ee2c9cd0b4be6fe9f0bc4116741" gracePeriod=45
Nov 03 14:25:42.408909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.408860    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused"
Nov 03 14:25:42.452288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.449203    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:42.752252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.750360    6481 generic.go:356] "Generic (PLEG): container finished" podID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerID="f34bde8e56935a6554b7148c33a6bf31b8c90ee2c9cd0b4be6fe9f0bc4116741" exitCode=0
Nov 03 14:25:42.752252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.750530    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" event={"ID":"61c4cd10-f303-4b2f-8208-41a0cb736312","Type":"ContainerDied","Data":"f34bde8e56935a6554b7148c33a6bf31b8c90ee2c9cd0b4be6fe9f0bc4116741"}
Nov 03 14:25:42.821265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.815401    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:25:42.914267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.913705    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm\") pod \"61c4cd10-f303-4b2f-8208-41a0cb736312\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") "
Nov 03 14:25:42.914267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.913930    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home\") pod \"61c4cd10-f303-4b2f-8208-41a0cb736312\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") "
Nov 03 14:25:42.914267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.914028    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs\" (UniqueName: \"kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs\") pod \"61c4cd10-f303-4b2f-8208-41a0cb736312\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") "
Nov 03 14:25:42.914267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.914082    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir\") pod \"61c4cd10-f303-4b2f-8208-41a0cb736312\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") "
Nov 03 14:25:42.914267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.914117    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache\") pod \"61c4cd10-f303-4b2f-8208-41a0cb736312\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") "
Nov 03 14:25:42.914267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.914174    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq\") pod \"61c4cd10-f303-4b2f-8208-41a0cb736312\" (UID: \"61c4cd10-f303-4b2f-8208-41a0cb736312\") "
Nov 03 14:25:42.915775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.915712    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home" pod "61c4cd10-f303-4b2f-8208-41a0cb736312" (UID: "61c4cd10-f303-4b2f-8208-41a0cb736312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:42.923956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.922686    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir" pod "61c4cd10-f303-4b2f-8208-41a0cb736312" (UID: "61c4cd10-f303-4b2f-8208-41a0cb736312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:42.923956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.922915    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache" pod "61c4cd10-f303-4b2f-8208-41a0cb736312" (UID: "61c4cd10-f303-4b2f-8208-41a0cb736312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:42.925667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.924971    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:42.925889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.925870    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:42.926078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.926062    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:42.941511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.941479    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:42.954689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.954404    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq" pod "61c4cd10-f303-4b2f-8208-41a0cb736312" (UID: "61c4cd10-f303-4b2f-8208-41a0cb736312"). InnerVolumeSpecName "kube-api-access-2kzqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:42.954689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.954489    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm" pod "61c4cd10-f303-4b2f-8208-41a0cb736312" (UID: "61c4cd10-f303-4b2f-8208-41a0cb736312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:25:42.954689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:42.954591    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs" pod "61c4cd10-f303-4b2f-8208-41a0cb736312" (UID: "61c4cd10-f303-4b2f-8208-41a0cb736312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:43.028334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.028190    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61c4cd10-f303-4b2f-8208-41a0cb736312-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:43.028334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.028256    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61c4cd10-f303-4b2f-8208-41a0cb736312-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:43.028334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.028273    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/61c4cd10-f303-4b2f-8208-41a0cb736312-kube-api-access-2kzqq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:43.784058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.784008    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2" event={"ID":"61c4cd10-f303-4b2f-8208-41a0cb736312","Type":"ContainerDied","Data":"4494bd0497de5d15ab12b551693862944f2b450b000553eb78ace05da729e94b"}
Nov 03 14:25:43.784630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.784609    6481 scope.go:122] "RemoveContainer" containerID="f34bde8e56935a6554b7148c33a6bf31b8c90ee2c9cd0b4be6fe9f0bc4116741"
Nov 03 14:25:43.784886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.784869    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"
Nov 03 14:25:43.901344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.891546    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"]
Nov 03 14:25:43.909265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:43.908922    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/auth-test-model-2d0da2-kserve-599ccc648f-f6sk2"]
Nov 03 14:25:44.125310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:44.103387    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" path="/var/lib/kubelet/pods/61c4cd10-f303-4b2f-8208-41a0cb736312/volumes"
Nov 03 14:25:44.333337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:44.328463    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:25:44.372365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:44.372317    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:44.412495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:44.411319    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:44.509943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:44.509348    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:44.831309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:44.827991    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:45.340064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.338317    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:45.340064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.339076    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:25:45.382335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.382297    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:45.383983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.382911    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:25:45.528265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.509330    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:45.528265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.509750    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:25:45.528265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.510559    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:45.528265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:45.519571    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:25:46.285262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.284919    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:25:46.335261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.332828    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:25:46.349249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.344286    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:25:46.386635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.386586    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:25:46.445570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.445537    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:25:46.925530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.925504    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:46.959685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.959653    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"]
Nov 03 14:25:46.960391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.960369    6481 cpu_manager.go:414] "containerMap: removing container" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main"
Nov 03 14:25:46.960391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.960391    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main"
Nov 03 14:25:46.960573 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.960565    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c4cd10-f303-4b2f-8208-41a0cb736312" containerName="main"
Nov 03 14:25:46.965427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.965406    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:46.982442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.982413    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:25:46.982591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.982555    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:25:46.982798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.982776    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:25:46.982869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.982857    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:25:46.984776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:46.984755    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"]
Nov 03 14:25:47.038712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.038683    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.038712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.038713    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.038907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.038749    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.140657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.140626    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.140814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.140668    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.140814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.140749    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.141355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.141335    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.142904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.142885    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.175224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.175201    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6\") pod \"maas-api-revoke-sub-2f99f44f-qpncc\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.278956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.278895    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:47.588950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:47.588919    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"]
Nov 03 14:25:47.590189 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:25:47.590150    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188f74e6_a0b8_4790_b0a2_0593f0f9c421.slice/crio-5bed1a94d37d37716d6e82f089b55ec94d27ce418770ed9f5949df7991ba1bb7 WatchSource:0}: Error finding container 5bed1a94d37d37716d6e82f089b55ec94d27ce418770ed9f5949df7991ba1bb7: Status 404 returned error can't find the container with id 5bed1a94d37d37716d6e82f089b55ec94d27ce418770ed9f5949df7991ba1bb7
Nov 03 14:25:48.083500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:48.083466    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc" event={"ID":"188f74e6-a0b8-4790-b0a2-0593f0f9c421","Type":"ContainerStarted","Data":"39e09bf352b788b60271bc4fa05fcd87c1cf0554d9979531fe61ed0cf1b67380"}
Nov 03 14:25:48.083678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:48.083523    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc" event={"ID":"188f74e6-a0b8-4790-b0a2-0593f0f9c421","Type":"ContainerStarted","Data":"5bed1a94d37d37716d6e82f089b55ec94d27ce418770ed9f5949df7991ba1bb7"}
Nov 03 14:25:48.127984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:48.127914    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc" podStartSLOduration=2.127895829 podStartE2EDuration="2.127895829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:48.125079675 +0000 UTC m=+826.175110859" watchObservedRunningTime="2026-11-03 14:25:48.127895829 +0000 UTC m=+826.177927014"
Nov 03 14:25:48.887916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:48.887825    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:49.095529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:49.095498    6481 generic.go:356] "Generic (PLEG): container finished" podID="188f74e6-a0b8-4790-b0a2-0593f0f9c421" containerID="39e09bf352b788b60271bc4fa05fcd87c1cf0554d9979531fe61ed0cf1b67380" exitCode=0
Nov 03 14:25:49.095700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:49.095577    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc" event={"ID":"188f74e6-a0b8-4790-b0a2-0593f0f9c421","Type":"ContainerDied","Data":"39e09bf352b788b60271bc4fa05fcd87c1cf0554d9979531fe61ed0cf1b67380"}
Nov 03 14:25:50.382307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.363480    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:50.515462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.514925    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca\") pod \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") "
Nov 03 14:25:50.516093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.515735    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6\") pod \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") "
Nov 03 14:25:50.516468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.516441    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca\") pod \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\" (UID: \"188f74e6-a0b8-4790-b0a2-0593f0f9c421\") "
Nov 03 14:25:50.517130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.517016    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca" pod "188f74e6-a0b8-4790-b0a2-0593f0f9c421" (UID: "188f74e6-a0b8-4790-b0a2-0593f0f9c421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:25:50.518039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.517990    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:50.636085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.636003    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca" pod "188f74e6-a0b8-4790-b0a2-0593f0f9c421" (UID: "188f74e6-a0b8-4790-b0a2-0593f0f9c421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:25:50.639398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.639367    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6" pod "188f74e6-a0b8-4790-b0a2-0593f0f9c421" (UID: "188f74e6-a0b8-4790-b0a2-0593f0f9c421"). InnerVolumeSpecName "kube-api-access-6c6p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:25:50.738381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.738306    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/188f74e6-a0b8-4790-b0a2-0593f0f9c421-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:50.738381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.738354    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6p6\" (UniqueName: \"kubernetes.io/projected/188f74e6-a0b8-4790-b0a2-0593f0f9c421-kube-api-access-6c6p6\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:25:50.927253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:50.918948    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:51.183452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:51.179525    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc" event={"ID":"188f74e6-a0b8-4790-b0a2-0593f0f9c421","Type":"ContainerDied","Data":"5bed1a94d37d37716d6e82f089b55ec94d27ce418770ed9f5949df7991ba1bb7"}
Nov 03 14:25:51.183452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:51.179590    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bed1a94d37d37716d6e82f089b55ec94d27ce418770ed9f5949df7991ba1bb7"
Nov 03 14:25:51.183452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:51.179658    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"
Nov 03 14:25:51.685606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:51.685569    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:25:51.698437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:51.698407    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:25:52.994576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:52.994170    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:54.946174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:54.946142    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:55.150473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:55.150442    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:57.114697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:57.114643    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:57.986636 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:25:57.986599    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:25:58.366598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:58.366523    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:25:59.205441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:25:59.205408    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:00.558019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.557988    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"]
Nov 03 14:26:00.558847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.558819    6481 cpu_manager.go:414] "containerMap: removing container" podUID="188f74e6-a0b8-4790-b0a2-0593f0f9c421" containerName="revoke-keys"
Nov 03 14:26:00.558847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.558847    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="188f74e6-a0b8-4790-b0a2-0593f0f9c421" containerName="revoke-keys"
Nov 03 14:26:00.559039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.559002    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="188f74e6-a0b8-4790-b0a2-0593f0f9c421" containerName="revoke-keys"
Nov 03 14:26:00.564608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.564585    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.584353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.584328    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"]
Nov 03 14:26:00.593779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.593751    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"rate-test-model-2d0da2-kserve-self-signed-certs\""
Nov 03 14:26:00.593779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.593771    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"default-dockercfg-fpknl\""
Nov 03 14:26:00.593968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.593877    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"openshift-service-ca.crt\""
Nov 03 14:26:00.594054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.594030    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-2d0da2\"/\"kube-root-ca.crt\""
Nov 03 14:26:00.600646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.600624    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.600801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.600782    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcfp\" (UniqueName: \"kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.600861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.600812    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.600972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.600948    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.601068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.601054    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.601107 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.601077    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.676293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.676266    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:00.702818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.702795    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.702947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.702826    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.702947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.702880    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.703076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.702967    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcfp\" (UniqueName: \"kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.703076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.702988    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.703076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.703045    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.703321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.703291    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.703441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.703417    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.703523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.703345    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.705427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.705406    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.705621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.705604    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.725607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.725586    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcfp\" (UniqueName: \"kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp\") pod \"rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") " pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.824204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.824108    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"]
Nov 03 14:26:00.835757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.831961    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:00.866312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.855016    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:26:00.866312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.856578    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:26:00.866312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.856693    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:26:00.866312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.856755    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:26:00.867738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.867709    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"]
Nov 03 14:26:00.880274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.880250    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:00.906248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.906202    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtvn9\" (UniqueName: \"kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:00.906586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.906562    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:00.906890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:00.906869    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.008661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.008623    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtvn9\" (UniqueName: \"kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.008847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.008703    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.008847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.008821    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.009478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.009454    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.011044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.011022    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.032220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.032197    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtvn9\" (UniqueName: \"kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9\") pod \"maas-api-revoke-sub-96e212dc-fqwmk\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.178780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.178749    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:01.201380 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:01.201335    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdd9d7fb_c03c_4814_9aee_6f3927d3e559.slice/crio-f3906108ba605e6b85f24a426d8ca99e1bfe2d7a51d43284d2395dd156bf9f56 WatchSource:0}: Error finding container f3906108ba605e6b85f24a426d8ca99e1bfe2d7a51d43284d2395dd156bf9f56: Status 404 returned error can't find the container with id f3906108ba605e6b85f24a426d8ca99e1bfe2d7a51d43284d2395dd156bf9f56
Nov 03 14:26:01.201890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.201770    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"]
Nov 03 14:26:01.314017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.313980    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" event={"ID":"bdd9d7fb-c03c-4814-9aee-6f3927d3e559","Type":"ContainerStarted","Data":"c33651932a7bacaf225c6ede35b7d47787b8485a4fc99118db6c251048b5be91"}
Nov 03 14:26:01.314267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.314036    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" event={"ID":"bdd9d7fb-c03c-4814-9aee-6f3927d3e559","Type":"ContainerStarted","Data":"f3906108ba605e6b85f24a426d8ca99e1bfe2d7a51d43284d2395dd156bf9f56"}
Nov 03 14:26:01.369281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.368176    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podStartSLOduration=1.368161365 podStartE2EDuration="1.368161365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:01.363198202 +0000 UTC m=+839.413229382" watchObservedRunningTime="2026-11-03 14:26:01.368161365 +0000 UTC m=+839.418192538"
Nov 03 14:26:01.388569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.388537    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:01.491270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.491240    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"]
Nov 03 14:26:01.712576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.712536    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:26:01.880748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:01.880708    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:02.327354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:02.327305    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk" event={"ID":"8efceb72-b62f-43ec-9e84-3c42f667746e","Type":"ContainerStarted","Data":"392c0d4a287d08a2831536f3f796f545a650e6cc850d0d6823bfd206d75bfcd5"}
Nov 03 14:26:02.327557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:02.327379    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk" event={"ID":"8efceb72-b62f-43ec-9e84-3c42f667746e","Type":"ContainerStarted","Data":"b28b4d46e872e18770cdcb8bd797452fbfa59d34ccd386ee4c31294da1f6cc2c"}
Nov 03 14:26:02.392331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:02.392264    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk" podStartSLOduration=2.392245339 podStartE2EDuration="2.392245339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:02.391050025 +0000 UTC m=+840.441081209" watchObservedRunningTime="2026-11-03 14:26:02.392245339 +0000 UTC m=+840.442276524"
Nov 03 14:26:02.513771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:02.513742    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:02.729589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:02.729494    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:03.346254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:03.338747    6481 generic.go:356] "Generic (PLEG): container finished" podID="8efceb72-b62f-43ec-9e84-3c42f667746e" containerID="392c0d4a287d08a2831536f3f796f545a650e6cc850d0d6823bfd206d75bfcd5" exitCode=0
Nov 03 14:26:03.346254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:03.340030    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk" event={"ID":"8efceb72-b62f-43ec-9e84-3c42f667746e","Type":"ContainerDied","Data":"392c0d4a287d08a2831536f3f796f545a650e6cc850d0d6823bfd206d75bfcd5"}
Nov 03 14:26:03.999271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:03.999153    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:04.507784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.507758    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:04.613032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.612948    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca\") pod \"8efceb72-b62f-43ec-9e84-3c42f667746e\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") "
Nov 03 14:26:04.613032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.612991    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9\" (UniqueName: \"kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9\") pod \"8efceb72-b62f-43ec-9e84-3c42f667746e\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") "
Nov 03 14:26:04.613281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.613166    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca\") pod \"8efceb72-b62f-43ec-9e84-3c42f667746e\" (UID: \"8efceb72-b62f-43ec-9e84-3c42f667746e\") "
Nov 03 14:26:04.613354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.613309    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca" pod "8efceb72-b62f-43ec-9e84-3c42f667746e" (UID: "8efceb72-b62f-43ec-9e84-3c42f667746e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:04.613702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.613682    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:04.615047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.615021    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9" pod "8efceb72-b62f-43ec-9e84-3c42f667746e" (UID: "8efceb72-b62f-43ec-9e84-3c42f667746e"). InnerVolumeSpecName "kube-api-access-gtvn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:04.615132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.615094    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca" pod "8efceb72-b62f-43ec-9e84-3c42f667746e" (UID: "8efceb72-b62f-43ec-9e84-3c42f667746e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:04.715000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.714961    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efceb72-b62f-43ec-9e84-3c42f667746e-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:04.715000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:04.714992    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtvn9\" (UniqueName: \"kubernetes.io/projected/8efceb72-b62f-43ec-9e84-3c42f667746e-kube-api-access-gtvn9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:05.356379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:05.356352    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"
Nov 03 14:26:05.356766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:05.356395    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk" event={"ID":"8efceb72-b62f-43ec-9e84-3c42f667746e","Type":"ContainerDied","Data":"b28b4d46e872e18770cdcb8bd797452fbfa59d34ccd386ee4c31294da1f6cc2c"}
Nov 03 14:26:05.356766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:05.356446    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28b4d46e872e18770cdcb8bd797452fbfa59d34ccd386ee4c31294da1f6cc2c"
Nov 03 14:26:06.261335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:06.261306    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:08.389063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:08.389024    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:10.402940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:10.402907    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:10.880733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:10.880705    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:10.893190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:10.893168    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:11.412796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:11.412770    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:12.424027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:12.423997    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:14.370789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.370756    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"]
Nov 03 14:26:14.371378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.371364    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8efceb72-b62f-43ec-9e84-3c42f667746e" containerName="revoke-keys"
Nov 03 14:26:14.371426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.371379    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efceb72-b62f-43ec-9e84-3c42f667746e" containerName="revoke-keys"
Nov 03 14:26:14.371507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.371497    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efceb72-b62f-43ec-9e84-3c42f667746e" containerName="revoke-keys"
Nov 03 14:26:14.376384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.376368    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.406968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.406939    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:26:14.407425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.407182    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:26:14.407425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.407308    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:26:14.407654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.407634    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:26:14.412058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.412036    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"]
Nov 03 14:26:14.504828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.504790    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:14.538557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.538525    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.538726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.538584    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.538726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.538666    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmk4\" (UniqueName: \"kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.639737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.639662    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmk4\" (UniqueName: \"kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.639881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.639811    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.639881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.639842    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.640527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.640501    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.642165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.642148    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.662814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.662792    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmk4\" (UniqueName: \"kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4\") pod \"maas-api-revoke-sub-e127a61f-7mg8w\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.689967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.689947    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:14.998550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:14.998520    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"]
Nov 03 14:26:14.999051 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:14.999023    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab1ed149_4286_4a8b_99f2_fafa27d52990.slice/crio-604b9577fa0dfafe6848f4547f7af363d6725aa9b09783bbd08ef112f06b1a76 WatchSource:0}: Error finding container 604b9577fa0dfafe6848f4547f7af363d6725aa9b09783bbd08ef112f06b1a76: Status 404 returned error can't find the container with id 604b9577fa0dfafe6848f4547f7af363d6725aa9b09783bbd08ef112f06b1a76
Nov 03 14:26:15.438791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:15.438761    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w" event={"ID":"ab1ed149-4286-4a8b-99f2-fafa27d52990","Type":"ContainerStarted","Data":"4edfce770acf089152140832d22379cacf7dcb932af891c21582a2cf4c8d7d87"}
Nov 03 14:26:15.439167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:15.438819    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w" event={"ID":"ab1ed149-4286-4a8b-99f2-fafa27d52990","Type":"ContainerStarted","Data":"604b9577fa0dfafe6848f4547f7af363d6725aa9b09783bbd08ef112f06b1a76"}
Nov 03 14:26:15.484499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:15.484452    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w" podStartSLOduration=1.484438372 podStartE2EDuration="1.484438372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:15.483105443 +0000 UTC m=+853.533136627" watchObservedRunningTime="2026-11-03 14:26:15.484438372 +0000 UTC m=+853.534469554"
Nov 03 14:26:16.447291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:16.447251    6481 generic.go:356] "Generic (PLEG): container finished" podID="ab1ed149-4286-4a8b-99f2-fafa27d52990" containerID="4edfce770acf089152140832d22379cacf7dcb932af891c21582a2cf4c8d7d87" exitCode=0
Nov 03 14:26:16.447694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:16.447306    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w" event={"ID":"ab1ed149-4286-4a8b-99f2-fafa27d52990","Type":"ContainerDied","Data":"4edfce770acf089152140832d22379cacf7dcb932af891c21582a2cf4c8d7d87"}
Nov 03 14:26:16.642186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:16.642152    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:16.752172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:16.752096    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:26:17.720019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.719972    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:17.761284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.761265    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:17.870673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.870637    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca\") pod \"ab1ed149-4286-4a8b-99f2-fafa27d52990\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") "
Nov 03 14:26:17.902254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.883522    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4\" (UniqueName: \"kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4\") pod \"ab1ed149-4286-4a8b-99f2-fafa27d52990\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") "
Nov 03 14:26:17.938252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.936016    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca\") pod \"ab1ed149-4286-4a8b-99f2-fafa27d52990\" (UID: \"ab1ed149-4286-4a8b-99f2-fafa27d52990\") "
Nov 03 14:26:17.938466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.916790    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca" pod "ab1ed149-4286-4a8b-99f2-fafa27d52990" (UID: "ab1ed149-4286-4a8b-99f2-fafa27d52990"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:17.938576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.925994    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4" pod "ab1ed149-4286-4a8b-99f2-fafa27d52990" (UID: "ab1ed149-4286-4a8b-99f2-fafa27d52990"). InnerVolumeSpecName "kube-api-access-qqmk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:17.940048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:17.940012    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca" pod "ab1ed149-4286-4a8b-99f2-fafa27d52990" (UID: "ab1ed149-4286-4a8b-99f2-fafa27d52990"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:18.056310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.052927    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmk4\" (UniqueName: \"kubernetes.io/projected/ab1ed149-4286-4a8b-99f2-fafa27d52990-kube-api-access-qqmk4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:18.056310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.052967    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:18.056310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.052984    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab1ed149-4286-4a8b-99f2-fafa27d52990-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:18.466298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.466261    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w" event={"ID":"ab1ed149-4286-4a8b-99f2-fafa27d52990","Type":"ContainerDied","Data":"604b9577fa0dfafe6848f4547f7af363d6725aa9b09783bbd08ef112f06b1a76"}
Nov 03 14:26:18.466298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.466285    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"
Nov 03 14:26:18.466501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.466316    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604b9577fa0dfafe6848f4547f7af363d6725aa9b09783bbd08ef112f06b1a76"
Nov 03 14:26:18.857885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:18.857805    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:19.073057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:19.073019    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:19.284314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:19.284274    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:19.517761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:19.517723    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:19.739657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:19.739624    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:21.563587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:21.563553    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:21.782728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:21.782692    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:22.430287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.430252    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:22.584265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.584237    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"]
Nov 03 14:26:22.584833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.584820    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ab1ed149-4286-4a8b-99f2-fafa27d52990" containerName="revoke-keys"
Nov 03 14:26:22.584884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.584834    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab1ed149-4286-4a8b-99f2-fafa27d52990" containerName="revoke-keys"
Nov 03 14:26:22.584951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.584942    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab1ed149-4286-4a8b-99f2-fafa27d52990" containerName="revoke-keys"
Nov 03 14:26:22.588652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.588636    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.607899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.607874    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\""
Nov 03 14:26:22.607899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.607889    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\""
Nov 03 14:26:22.608066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.607906    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\""
Nov 03 14:26:22.620792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.620769    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\""
Nov 03 14:26:22.620997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.620804    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\""
Nov 03 14:26:22.620997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.620909    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\""
Nov 03 14:26:22.623866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.623837    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.623994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.623967    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.624044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.624012    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdpr\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.624082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.624055    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.627111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.627089    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"]
Nov 03 14:26:22.725852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.725781    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.725852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.725846    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.726061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.725952    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdpr\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.726061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.725991    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.726540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.726514    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.726729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.726712    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.728535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.728512    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.786988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.786918    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdpr\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:22.903163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:22.903127    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:23.213726 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:23.213694    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9878d82f_ee00_4f1c_8a07_035deb8750ad.slice/crio-c0170fc3bc7ff8fe7b21714a58cecbf001cd195fc88820a41a4d99c5af1093e5 WatchSource:0}: Error finding container c0170fc3bc7ff8fe7b21714a58cecbf001cd195fc88820a41a4d99c5af1093e5: Status 404 returned error can't find the container with id c0170fc3bc7ff8fe7b21714a58cecbf001cd195fc88820a41a4d99c5af1093e5
Nov 03 14:26:23.213972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:23.213950    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"]
Nov 03 14:26:23.508335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:23.508256    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w" event={"ID":"9878d82f-ee00-4f1c-8a07-035deb8750ad","Type":"ContainerStarted","Data":"6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7"}
Nov 03 14:26:23.508479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:23.508336    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w" event={"ID":"9878d82f-ee00-4f1c-8a07-035deb8750ad","Type":"ContainerStarted","Data":"c0170fc3bc7ff8fe7b21714a58cecbf001cd195fc88820a41a4d99c5af1093e5"}
Nov 03 14:26:23.508479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:23.508373    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:23.557681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:23.557621    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w" podStartSLOduration=1.557605815 podStartE2EDuration="1.557605815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:23.554187847 +0000 UTC m=+861.604219033" watchObservedRunningTime="2026-11-03 14:26:23.557605815 +0000 UTC m=+861.607636998"
Nov 03 14:26:23.904304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:23.904164    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:24.812977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.812944    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"]
Nov 03 14:26:24.817246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.817212    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.840349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.840326    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"]
Nov 03 14:26:24.848960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.848938    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:26:24.849107 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.849090    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:26:24.849181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.849129    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:26:24.849181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.849136    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:26:24.851537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.851511    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.851706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.851686    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.851834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.851816    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64dj\" (UniqueName: \"kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.953441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.953391    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64dj\" (UniqueName: \"kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.953829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.953530    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.953829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.953662    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.954178 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.954159    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.955933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.955912    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:24.976079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:24.976050    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64dj\" (UniqueName: \"kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj\") pod \"maas-api-revoke-sub-aa080eb0-2tdt6\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:25.131111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:25.131039    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:25.442718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:25.440284    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"]
Nov 03 14:26:25.523846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:25.523819    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6" event={"ID":"6ffdb16d-6147-4da6-b43f-ddea11bd00ec","Type":"ContainerStarted","Data":"01f37538ea0fb3471a537b658d16e153babae7503fd30a6849ffd4a2b294b210"}
Nov 03 14:26:26.013194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:26.013120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:26.537537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:26.532606    6481 generic.go:356] "Generic (PLEG): container finished" podID="6ffdb16d-6147-4da6-b43f-ddea11bd00ec" containerID="73b77c3ffb4c0ff785c2406140e924e447bd4b91bd3f4f4b0842483423774589" exitCode=0
Nov 03 14:26:26.537537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:26.532703    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6" event={"ID":"6ffdb16d-6147-4da6-b43f-ddea11bd00ec","Type":"ContainerDied","Data":"73b77c3ffb4c0ff785c2406140e924e447bd4b91bd3f4f4b0842483423774589"}
Nov 03 14:26:27.057022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.056938    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"]
Nov 03 14:26:27.063584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.063560    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-691d57b2-8z2qd"]
Nov 03 14:26:27.692135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.692110    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:27.790329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.790216    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca\") pod \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") "
Nov 03 14:26:27.790522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.790436    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca\") pod \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") "
Nov 03 14:26:27.790727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.790689    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca" pod "6ffdb16d-6147-4da6-b43f-ddea11bd00ec" (UID: "6ffdb16d-6147-4da6-b43f-ddea11bd00ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:27.790895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.790871    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj\" (UniqueName: \"kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj\") pod \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\" (UID: \"6ffdb16d-6147-4da6-b43f-ddea11bd00ec\") "
Nov 03 14:26:27.791540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.791515    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:27.793220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.793180    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj" pod "6ffdb16d-6147-4da6-b43f-ddea11bd00ec" (UID: "6ffdb16d-6147-4da6-b43f-ddea11bd00ec"). InnerVolumeSpecName "kube-api-access-m64dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:27.793609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.793583    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca" pod "6ffdb16d-6147-4da6-b43f-ddea11bd00ec" (UID: "6ffdb16d-6147-4da6-b43f-ddea11bd00ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:27.892652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.892609    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:27.892652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:27.892655    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m64dj\" (UniqueName: \"kubernetes.io/projected/6ffdb16d-6147-4da6-b43f-ddea11bd00ec-kube-api-access-m64dj\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:28.002820 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:26:28.002781    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:26:28.010191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:28.010154    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:28.078165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:28.078069    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c062733e-0431-47ee-a345-d74e51932672" path="/var/lib/kubelet/pods/c062733e-0431-47ee-a345-d74e51932672/volumes"
Nov 03 14:26:28.551056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:28.551016    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6" event={"ID":"6ffdb16d-6147-4da6-b43f-ddea11bd00ec","Type":"ContainerDied","Data":"01f37538ea0fb3471a537b658d16e153babae7503fd30a6849ffd4a2b294b210"}
Nov 03 14:26:28.551056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:28.551041    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"
Nov 03 14:26:28.551284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:28.551075    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01f37538ea0fb3471a537b658d16e153babae7503fd30a6849ffd4a2b294b210"
Nov 03 14:26:30.130505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:30.130453    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:31.913256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:31.905322    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:32.287583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:32.287489    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:32.336457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:32.335304    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:32.410383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:32.410330    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:32.427362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:32.427311    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:26:32.448370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:32.448318    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:33.212253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.195689    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:33.294253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.289592    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:26:33.294253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.290480    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:26:33.346280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.342638    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:33.346280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.343501    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:26:33.479193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.439871    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:33.487397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.480578    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:33.493591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.483897    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:26:33.496301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.488014    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:26:33.496962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.496943    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:33.497156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.497141    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:26:33.793306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.793132    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:26:33.899852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.899823    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:26:33.924538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:33.919590    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:26:34.005031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:34.005000    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:26:34.024132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:34.022279    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:26:34.518736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:34.518702    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:26:35.220404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:35.220370    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:36.032078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:36.032049    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:36.242461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:36.242433    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:37.301067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.301037    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:37.888703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.888670    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"]
Nov 03 14:26:37.889331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.889316    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6ffdb16d-6147-4da6-b43f-ddea11bd00ec" containerName="revoke-keys"
Nov 03 14:26:37.889383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.889333    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffdb16d-6147-4da6-b43f-ddea11bd00ec" containerName="revoke-keys"
Nov 03 14:26:37.889443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.889435    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffdb16d-6147-4da6-b43f-ddea11bd00ec" containerName="revoke-keys"
Nov 03 14:26:37.894511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.894491    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:37.918276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.918243    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"]
Nov 03 14:26:37.925760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.925730    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:26:37.925935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.925763    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:26:37.926053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.925888    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:26:37.926161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.925931    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:26:37.967037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.967012    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:37.967176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.967059    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:37.967176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:37.967139    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmjf6\" (UniqueName: \"kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.068972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.068935    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.069157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.069000    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.069157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.069051    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmjf6\" (UniqueName: \"kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.069923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.069888    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.071582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.071553    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.091104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.091084    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmjf6\" (UniqueName: \"kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6\") pod \"maas-api-revoke-sub-17522a21-vzt24\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.208602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.208571    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:38.518565 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:38.518521    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5111e93b_6a39_49ea_a56b_fa243e84011a.slice/crio-9e23ea3c4e0b5c429563457e15eb690fb01034539f56e5b574484c1725a5eb55 WatchSource:0}: Error finding container 9e23ea3c4e0b5c429563457e15eb690fb01034539f56e5b574484c1725a5eb55: Status 404 returned error can't find the container with id 9e23ea3c4e0b5c429563457e15eb690fb01034539f56e5b574484c1725a5eb55
Nov 03 14:26:38.519056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.519038    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"]
Nov 03 14:26:38.862895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.862866    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24" event={"ID":"5111e93b-6a39-49ea-a56b-fa243e84011a","Type":"ContainerStarted","Data":"6671ab8156e68b9e97b262957a5a1a757aa3987758b56e02c75f625f71debb5a"}
Nov 03 14:26:38.863031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.862924    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24" event={"ID":"5111e93b-6a39-49ea-a56b-fa243e84011a","Type":"ContainerStarted","Data":"9e23ea3c4e0b5c429563457e15eb690fb01034539f56e5b574484c1725a5eb55"}
Nov 03 14:26:38.924137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:38.924076    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24" podStartSLOduration=1.924057773 podStartE2EDuration="1.924057773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:38.906614498 +0000 UTC m=+876.956645682" watchObservedRunningTime="2026-11-03 14:26:38.924057773 +0000 UTC m=+876.974088958"
Nov 03 14:26:39.358109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:39.358058    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:39.879633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:39.879597    6481 generic.go:356] "Generic (PLEG): container finished" podID="5111e93b-6a39-49ea-a56b-fa243e84011a" containerID="6671ab8156e68b9e97b262957a5a1a757aa3987758b56e02c75f625f71debb5a" exitCode=0
Nov 03 14:26:39.880047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:39.879688    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24" event={"ID":"5111e93b-6a39-49ea-a56b-fa243e84011a","Type":"ContainerDied","Data":"6671ab8156e68b9e97b262957a5a1a757aa3987758b56e02c75f625f71debb5a"}
Nov 03 14:26:41.043546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.043518    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:41.120193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.120146    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca\") pod \"5111e93b-6a39-49ea-a56b-fa243e84011a\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") "
Nov 03 14:26:41.120395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.120368    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca\") pod \"5111e93b-6a39-49ea-a56b-fa243e84011a\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") "
Nov 03 14:26:41.120667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.120594    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6\" (UniqueName: \"kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6\") pod \"5111e93b-6a39-49ea-a56b-fa243e84011a\" (UID: \"5111e93b-6a39-49ea-a56b-fa243e84011a\") "
Nov 03 14:26:41.120842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.120816    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca" pod "5111e93b-6a39-49ea-a56b-fa243e84011a" (UID: "5111e93b-6a39-49ea-a56b-fa243e84011a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:41.121216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.121193    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:41.122402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.122370    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca" pod "5111e93b-6a39-49ea-a56b-fa243e84011a" (UID: "5111e93b-6a39-49ea-a56b-fa243e84011a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:41.123102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.123080    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6" pod "5111e93b-6a39-49ea-a56b-fa243e84011a" (UID: "5111e93b-6a39-49ea-a56b-fa243e84011a"). InnerVolumeSpecName "kube-api-access-gmjf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:41.156002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.155946    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"]
Nov 03 14:26:41.156537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.156523    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5111e93b-6a39-49ea-a56b-fa243e84011a" containerName="revoke-keys"
Nov 03 14:26:41.156584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.156538    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5111e93b-6a39-49ea-a56b-fa243e84011a" containerName="revoke-keys"
Nov 03 14:26:41.156673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.156664    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5111e93b-6a39-49ea-a56b-fa243e84011a" containerName="revoke-keys"
Nov 03 14:26:41.160743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.160725    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.184251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.184196    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"]
Nov 03 14:26:41.222930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.222903    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.223089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.222971    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.223145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.223116    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xvb\" (UniqueName: \"kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.223307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.223292    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmjf6\" (UniqueName: \"kubernetes.io/projected/5111e93b-6a39-49ea-a56b-fa243e84011a-kube-api-access-gmjf6\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:41.223365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.223310    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111e93b-6a39-49ea-a56b-fa243e84011a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:41.324742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.324715    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.324905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.324779    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.324905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.324842    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xvb\" (UniqueName: \"kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.325490 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.325468    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.327030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.327013    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.348920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.348897    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xvb\" (UniqueName: \"kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb\") pod \"maas-api-revoke-sub-b1c71c5c-4zlvk\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.413685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.413654    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:41.474677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.474651    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:41.781422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.781383    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"]
Nov 03 14:26:41.781600 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:41.781571    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731308c2_51f5_438c_9d15_ccb69799717a.slice/crio-6bc9654b8dcfa34e1549d664273ea579e08cca6c573960b1802d2e743952ed2e WatchSource:0}: Error finding container 6bc9654b8dcfa34e1549d664273ea579e08cca6c573960b1802d2e743952ed2e: Status 404 returned error can't find the container with id 6bc9654b8dcfa34e1549d664273ea579e08cca6c573960b1802d2e743952ed2e
Nov 03 14:26:41.894766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.894742    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"
Nov 03 14:26:41.894938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.894776    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24" event={"ID":"5111e93b-6a39-49ea-a56b-fa243e84011a","Type":"ContainerDied","Data":"9e23ea3c4e0b5c429563457e15eb690fb01034539f56e5b574484c1725a5eb55"}
Nov 03 14:26:41.894938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.894825    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e23ea3c4e0b5c429563457e15eb690fb01034539f56e5b574484c1725a5eb55"
Nov 03 14:26:41.896348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.896322    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk" event={"ID":"731308c2-51f5-438c-9d15-ccb69799717a","Type":"ContainerStarted","Data":"6bc9654b8dcfa34e1549d664273ea579e08cca6c573960b1802d2e743952ed2e"}
Nov 03 14:26:41.983989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:41.983952    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:26:42.971265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:42.967733    6481 generic.go:356] "Generic (PLEG): container finished" podID="731308c2-51f5-438c-9d15-ccb69799717a" containerID="f799a5bcbc6f65cf00f1645d2378ebfd61501ff97b1d6155f859fb03f1a12c7a" exitCode=0
Nov 03 14:26:42.971265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:42.967970    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk" event={"ID":"731308c2-51f5-438c-9d15-ccb69799717a","Type":"ContainerDied","Data":"f799a5bcbc6f65cf00f1645d2378ebfd61501ff97b1d6155f859fb03f1a12c7a"}
Nov 03 14:26:43.929254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:43.919309    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:26:44.288768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:44.288662    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:45.159302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.157568    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:45.160783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.160690    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk" event={"ID":"731308c2-51f5-438c-9d15-ccb69799717a","Type":"ContainerDied","Data":"6bc9654b8dcfa34e1549d664273ea579e08cca6c573960b1802d2e743952ed2e"}
Nov 03 14:26:45.160783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.160751    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc9654b8dcfa34e1549d664273ea579e08cca6c573960b1802d2e743952ed2e"
Nov 03 14:26:45.257250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.253948    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb\" (UniqueName: \"kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb\") pod \"731308c2-51f5-438c-9d15-ccb69799717a\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") "
Nov 03 14:26:45.264312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.257523    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca\") pod \"731308c2-51f5-438c-9d15-ccb69799717a\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") "
Nov 03 14:26:45.268247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.264691    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca\") pod \"731308c2-51f5-438c-9d15-ccb69799717a\" (UID: \"731308c2-51f5-438c-9d15-ccb69799717a\") "
Nov 03 14:26:45.279011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.267377    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca" pod "731308c2-51f5-438c-9d15-ccb69799717a" (UID: "731308c2-51f5-438c-9d15-ccb69799717a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:45.279218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.279192    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:45.426334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.424411    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb" pod "731308c2-51f5-438c-9d15-ccb69799717a" (UID: "731308c2-51f5-438c-9d15-ccb69799717a"). InnerVolumeSpecName "kube-api-access-f4xvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:45.453269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.431362    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xvb\" (UniqueName: \"kubernetes.io/projected/731308c2-51f5-438c-9d15-ccb69799717a-kube-api-access-f4xvb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:45.453269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.441329    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca" pod "731308c2-51f5-438c-9d15-ccb69799717a" (UID: "731308c2-51f5-438c-9d15-ccb69799717a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:45.576269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:45.570570    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731308c2-51f5-438c-9d15-ccb69799717a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:46.167723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:46.167694    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"
Nov 03 14:26:46.517027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:46.516993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:48.263271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.263225    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"]
Nov 03 14:26:48.263902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.263886    6481 cpu_manager.go:414] "containerMap: removing container" podUID="731308c2-51f5-438c-9d15-ccb69799717a" containerName="revoke-keys"
Nov 03 14:26:48.263957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.263904    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731308c2-51f5-438c-9d15-ccb69799717a" containerName="revoke-keys"
Nov 03 14:26:48.264026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.264016    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="731308c2-51f5-438c-9d15-ccb69799717a" containerName="revoke-keys"
Nov 03 14:26:48.267460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.267445    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.289979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.289952    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"]
Nov 03 14:26:48.297878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.297856    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:26:48.298026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.298009    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:26:48.298146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.297926    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:26:48.298202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.298008    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:26:48.315040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.315010    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.315156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.315058    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw6xw\" (UniqueName: \"kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.315251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.315208    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.416538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.416508    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.416675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.416546    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw6xw\" (UniqueName: \"kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.416675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.416580    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.417327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.417300    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.418783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.418763    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.439370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.439349    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw6xw\" (UniqueName: \"kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw\") pod \"maas-api-revoke-sub-48e78d10-7trzv\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.580900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.580818    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:48.768221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.768187    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:48.892010 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:48.891976    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a63e8e8_5fbd_4a9f_8020_072fdd5fb542.slice/crio-5ddece3fdcc49e93364934130763249739d64034718aba6d3c1796fabaf04b41 WatchSource:0}: Error finding container 5ddece3fdcc49e93364934130763249739d64034718aba6d3c1796fabaf04b41: Status 404 returned error can't find the container with id 5ddece3fdcc49e93364934130763249739d64034718aba6d3c1796fabaf04b41
Nov 03 14:26:48.892162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:48.892133    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"]
Nov 03 14:26:49.220796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:49.220765    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv" event={"ID":"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542","Type":"ContainerStarted","Data":"d7cf71d9071a686c290814ff0a6607a1b012c1f4d5e32bf257fbdccf676502f4"}
Nov 03 14:26:49.220917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:49.220831    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv" event={"ID":"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542","Type":"ContainerStarted","Data":"5ddece3fdcc49e93364934130763249739d64034718aba6d3c1796fabaf04b41"}
Nov 03 14:26:49.270424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:49.270377    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv" podStartSLOduration=1.270365009 podStartE2EDuration="1.270365009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:49.265921044 +0000 UTC m=+887.315952227" watchObservedRunningTime="2026-11-03 14:26:49.270365009 +0000 UTC m=+887.320396191"
Nov 03 14:26:50.230069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:50.230037    6481 generic.go:356] "Generic (PLEG): container finished" podID="9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" containerID="d7cf71d9071a686c290814ff0a6607a1b012c1f4d5e32bf257fbdccf676502f4" exitCode=0
Nov 03 14:26:50.230277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:50.230124    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv" event={"ID":"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542","Type":"ContainerDied","Data":"d7cf71d9071a686c290814ff0a6607a1b012c1f4d5e32bf257fbdccf676502f4"}
Nov 03 14:26:50.928287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:50.928222    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:51.391538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.391516    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:51.454214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.454178    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca\") pod \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") "
Nov 03 14:26:51.454386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.454241    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw\" (UniqueName: \"kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw\") pod \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") "
Nov 03 14:26:51.454463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.454441    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca\") pod \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\" (UID: \"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542\") "
Nov 03 14:26:51.454844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.454813    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca" pod "9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" (UID: "9a63e8e8-5fbd-4a9f-8020-072fdd5fb542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:51.455026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.455001    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:51.456807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.456777    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca" pod "9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" (UID: "9a63e8e8-5fbd-4a9f-8020-072fdd5fb542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:51.456916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.456894    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw" pod "9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" (UID: "9a63e8e8-5fbd-4a9f-8020-072fdd5fb542"). InnerVolumeSpecName "kube-api-access-lw6xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:51.555979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.555905    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:51.555979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:51.555934    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw6xw\" (UniqueName: \"kubernetes.io/projected/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542-kube-api-access-lw6xw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:52.253895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:52.253840    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"
Nov 03 14:26:52.253895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:52.253861    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv" event={"ID":"9a63e8e8-5fbd-4a9f-8020-072fdd5fb542","Type":"ContainerDied","Data":"5ddece3fdcc49e93364934130763249739d64034718aba6d3c1796fabaf04b41"}
Nov 03 14:26:52.254346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:52.253949    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ddece3fdcc49e93364934130763249739d64034718aba6d3c1796fabaf04b41"
Nov 03 14:26:52.319495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:52.319462    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:53.002394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.002363    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"]
Nov 03 14:26:53.002983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.002969    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" containerName="revoke-keys"
Nov 03 14:26:53.003037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.002984    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" containerName="revoke-keys"
Nov 03 14:26:53.003095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.003086    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" containerName="revoke-keys"
Nov 03 14:26:53.006690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.006672    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.039584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.039552    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:26:53.039733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.039655    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:26:53.039928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.039913    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:26:53.040291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.040272    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:26:53.044458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.044433    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"]
Nov 03 14:26:53.075517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.075491    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.075639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.075624    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.075749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.075730    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.177055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.177023    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.177212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.177124    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.177212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.177168    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.177923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.177893    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.180283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.180250    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.200847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.200825    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s\") pod \"maas-api-revoke-sub-89d34580-dvctc\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.308095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.307991    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:53.324066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.324037    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:53.635127 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:26:53.635086    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7599554e_f0ee_43c3_ac31_5ca551d718ce.slice/crio-0f13e4e14a85aa7a12f5800690bf01284f563fbcc1a897f59e703bd57dd48079 WatchSource:0}: Error finding container 0f13e4e14a85aa7a12f5800690bf01284f563fbcc1a897f59e703bd57dd48079: Status 404 returned error can't find the container with id 0f13e4e14a85aa7a12f5800690bf01284f563fbcc1a897f59e703bd57dd48079
Nov 03 14:26:53.639602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:53.639584    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"]
Nov 03 14:26:54.275353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:54.275257    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc" event={"ID":"7599554e-f0ee-43c3-ac31-5ca551d718ce","Type":"ContainerStarted","Data":"bcaaba40678622f9637d54d894198c951cba70f2a1e8a238277db48a12d6216b"}
Nov 03 14:26:54.275353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:54.275311    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc" event={"ID":"7599554e-f0ee-43c3-ac31-5ca551d718ce","Type":"ContainerStarted","Data":"0f13e4e14a85aa7a12f5800690bf01284f563fbcc1a897f59e703bd57dd48079"}
Nov 03 14:26:54.324485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:54.324434    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc" podStartSLOduration=2.324421391 podStartE2EDuration="2.324421391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:26:54.320907967 +0000 UTC m=+892.370939149" watchObservedRunningTime="2026-11-03 14:26:54.324421391 +0000 UTC m=+892.374452582"
Nov 03 14:26:55.283360 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:55.283326    6481 generic.go:356] "Generic (PLEG): container finished" podID="7599554e-f0ee-43c3-ac31-5ca551d718ce" containerID="bcaaba40678622f9637d54d894198c951cba70f2a1e8a238277db48a12d6216b" exitCode=0
Nov 03 14:26:55.283521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:55.283363    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc" event={"ID":"7599554e-f0ee-43c3-ac31-5ca551d718ce","Type":"ContainerDied","Data":"bcaaba40678622f9637d54d894198c951cba70f2a1e8a238277db48a12d6216b"}
Nov 03 14:26:55.422243 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:55.422197    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:56.979090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:56.978709    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:57.103250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.102304    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s\") pod \"7599554e-f0ee-43c3-ac31-5ca551d718ce\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") "
Nov 03 14:26:57.103250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.102446    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca\") pod \"7599554e-f0ee-43c3-ac31-5ca551d718ce\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") "
Nov 03 14:26:57.103250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.102532    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca\") pod \"7599554e-f0ee-43c3-ac31-5ca551d718ce\" (UID: \"7599554e-f0ee-43c3-ac31-5ca551d718ce\") "
Nov 03 14:26:57.113131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.113054    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca" pod "7599554e-f0ee-43c3-ac31-5ca551d718ce" (UID: "7599554e-f0ee-43c3-ac31-5ca551d718ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:26:57.132478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.132443    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s" pod "7599554e-f0ee-43c3-ac31-5ca551d718ce" (UID: "7599554e-f0ee-43c3-ac31-5ca551d718ce"). InnerVolumeSpecName "kube-api-access-m2r8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:26:57.141410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.141349    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca" pod "7599554e-f0ee-43c3-ac31-5ca551d718ce" (UID: "7599554e-f0ee-43c3-ac31-5ca551d718ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:26:57.205397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.205362    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:57.206141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.206116    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2r8s\" (UniqueName: \"kubernetes.io/projected/7599554e-f0ee-43c3-ac31-5ca551d718ce-kube-api-access-m2r8s\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:57.206319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.206303    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7599554e-f0ee-43c3-ac31-5ca551d718ce-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:26:57.355252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.354983    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc" event={"ID":"7599554e-f0ee-43c3-ac31-5ca551d718ce","Type":"ContainerDied","Data":"0f13e4e14a85aa7a12f5800690bf01284f563fbcc1a897f59e703bd57dd48079"}
Nov 03 14:26:57.355252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.355051    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f13e4e14a85aa7a12f5800690bf01284f563fbcc1a897f59e703bd57dd48079"
Nov 03 14:26:57.355252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.355127    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"
Nov 03 14:26:57.507251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:57.503607    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:58.154302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:26:58.150637    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:26:58.170941 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:26:58.170846    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:27:00.172211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:00.172173    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:02.207139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:02.207103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:03.228746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.228713    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"]
Nov 03 14:27:03.229569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.229545    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7599554e-f0ee-43c3-ac31-5ca551d718ce" containerName="revoke-keys"
Nov 03 14:27:03.229686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.229569    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7599554e-f0ee-43c3-ac31-5ca551d718ce" containerName="revoke-keys"
Nov 03 14:27:03.229763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.229750    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7599554e-f0ee-43c3-ac31-5ca551d718ce" containerName="revoke-keys"
Nov 03 14:27:03.238142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.238117    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.256694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.256662    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"]
Nov 03 14:27:03.264117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.264081    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:27:03.264372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.264348    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:27:03.264544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.264524    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:27:03.267400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.267382    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:27:03.376338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.376307    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.376476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.376354    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.376516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.376487    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnv4z\" (UniqueName: \"kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.477932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.477897    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnv4z\" (UniqueName: \"kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.478159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.478139    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.478218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.478201    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.478807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.478745    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.480361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.480338    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.501607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.501588    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnv4z\" (UniqueName: \"kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z\") pod \"maas-api-revoke-sub-cc0e052c-pdzz2\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.556963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.556942    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:03.865538 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:03.865507    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0923ce12_8c7e_43fb_9aee_0f1b26e4ea20.slice/crio-5c777b9a0d1c0ab50aecc6109494e93a67a27939b6cc78172b33700961636bf5 WatchSource:0}: Error finding container 5c777b9a0d1c0ab50aecc6109494e93a67a27939b6cc78172b33700961636bf5: Status 404 returned error can't find the container with id 5c777b9a0d1c0ab50aecc6109494e93a67a27939b6cc78172b33700961636bf5
Nov 03 14:27:03.865811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:03.865786    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"]
Nov 03 14:27:04.323146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:04.323115    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:04.410397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:04.410365    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2" event={"ID":"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20","Type":"ContainerStarted","Data":"a37b1cd2c4b3630a34be9c180fe9cd09dbea3d6f4ec1f34d9051d09fb0197f13"}
Nov 03 14:27:04.410561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:04.410421    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2" event={"ID":"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20","Type":"ContainerStarted","Data":"5c777b9a0d1c0ab50aecc6109494e93a67a27939b6cc78172b33700961636bf5"}
Nov 03 14:27:04.458906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:04.458860    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2" podStartSLOduration=1.458846417 podStartE2EDuration="1.458846417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:04.456257989 +0000 UTC m=+902.506289172" watchObservedRunningTime="2026-11-03 14:27:04.458846417 +0000 UTC m=+902.508877654"
Nov 03 14:27:05.220100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:05.218898    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:05.420467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:05.420432    6481 generic.go:356] "Generic (PLEG): container finished" podID="0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" containerID="a37b1cd2c4b3630a34be9c180fe9cd09dbea3d6f4ec1f34d9051d09fb0197f13" exitCode=0
Nov 03 14:27:05.420841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:05.420516    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2" event={"ID":"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20","Type":"ContainerDied","Data":"a37b1cd2c4b3630a34be9c180fe9cd09dbea3d6f4ec1f34d9051d09fb0197f13"}
Nov 03 14:27:05.653325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:05.653245    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:06.515340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.515304    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:06.597212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.597189    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:06.718658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.718588    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca\") pod \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") "
Nov 03 14:27:06.718658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.718643    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z\" (UniqueName: \"kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z\") pod \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") "
Nov 03 14:27:06.718873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.718686    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca\") pod \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\" (UID: \"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20\") "
Nov 03 14:27:06.719130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.719108    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca" pod "0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" (UID: "0923ce12-8c7e-43fb-9aee-0f1b26e4ea20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:06.720704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.720682    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca" pod "0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" (UID: "0923ce12-8c7e-43fb-9aee-0f1b26e4ea20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:27:06.720797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.720774    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z" pod "0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" (UID: "0923ce12-8c7e-43fb-9aee-0f1b26e4ea20"). InnerVolumeSpecName "kube-api-access-qnv4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:06.819834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.819807    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnv4z\" (UniqueName: \"kubernetes.io/projected/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-kube-api-access-qnv4z\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:06.819834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.819836    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:06.819976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:06.819850    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:07.435457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:07.435427    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"
Nov 03 14:27:07.435622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:07.435468    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2" event={"ID":"0923ce12-8c7e-43fb-9aee-0f1b26e4ea20","Type":"ContainerDied","Data":"5c777b9a0d1c0ab50aecc6109494e93a67a27939b6cc78172b33700961636bf5"}
Nov 03 14:27:07.435622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:07.435524    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c777b9a0d1c0ab50aecc6109494e93a67a27939b6cc78172b33700961636bf5"
Nov 03 14:27:08.644415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:08.644382    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:27:08.722320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:08.722278    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:10.942253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:10.939516    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:27:11.014263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.013319    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:11.039253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.027301    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:11.896284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.892588    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:11.917540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.917493    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:27:11.989758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.989696    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:11.991291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.991270    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:11.995288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:11.995202    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:12.024249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:12.019357    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:12.024249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:12.020028    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:12.734184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:12.734155    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:12.798049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:12.798012    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:12.813545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:12.813518    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:13.149025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:13.148919    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:14.589679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:14.589639    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:15.801313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.801273    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"]
Nov 03 14:27:15.801888 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.801874    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" containerName="revoke-keys"
Nov 03 14:27:15.801940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.801889    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" containerName="revoke-keys"
Nov 03 14:27:15.802012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.802002    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" containerName="revoke-keys"
Nov 03 14:27:15.806637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.806620    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.824436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.824409    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czn8\" (UniqueName: \"kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.824551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.824515    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.824551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.824538    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.832564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.832534    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"]
Nov 03 14:27:15.839450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.839425    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:27:15.839568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.839480    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:27:15.913552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.913528    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:27:15.913747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.913733    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:27:15.925427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.925403    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czn8\" (UniqueName: \"kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.925514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.925479    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.925514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.925496    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.926123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.926105    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.927813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.927795    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:15.953567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:15.953547    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czn8\" (UniqueName: \"kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8\") pod \"maas-api-revoke-sub-d27f21ec-r6p6n\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:16.053836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.053762    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"]
Nov 03 14:27:16.058332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.058316    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.087039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.087013    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"]
Nov 03 14:27:16.119982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.119953    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:16.121108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.121074    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:16.128636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.128609    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpmct\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.128757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.128650    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.128757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.128736    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.128853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.128839    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.128889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.128880    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.128988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.128969    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.129023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.129005    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.129059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.129044    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.129101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.129074    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231190    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231276    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231453    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231516    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231609    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231640    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231664    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231707    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.231860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231838    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpmct\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.232424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.231907    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.232424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.232136    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.232424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.232370    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.233280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.233220    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.233411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.233315    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.234695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.234670    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.238072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.238048    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.259791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.259692    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpmct\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.265176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.263679    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token\") pod \"e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") " pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.377762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.377454    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:16.396176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.394776    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:16.443472 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:16.443444    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8bde37_0b06_46a8_a187_3569ae35dcbc.slice/crio-8bba06536dae886347f5cb2b621a0bab33c29341f878d293ca853f93b5119a0e WatchSource:0}: Error finding container 8bba06536dae886347f5cb2b621a0bab33c29341f878d293ca853f93b5119a0e: Status 404 returned error can't find the container with id 8bba06536dae886347f5cb2b621a0bab33c29341f878d293ca853f93b5119a0e
Nov 03 14:27:16.445877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.445847    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"]
Nov 03 14:27:16.610536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.610505    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:16.704121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.704092    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n" event={"ID":"ae8bde37-0b06-46a8-a187-3569ae35dcbc","Type":"ContainerStarted","Data":"8bba06536dae886347f5cb2b621a0bab33c29341f878d293ca853f93b5119a0e"}
Nov 03 14:27:16.738955 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:16.738926    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9953e53b_bb90_4fad_af85_07532f02339f.slice/crio-b1b577d249df5a7e808bd0279199d930c9682a4f770bf165d8e1b1f5e4814e8e WatchSource:0}: Error finding container b1b577d249df5a7e808bd0279199d930c9682a4f770bf165d8e1b1f5e4814e8e: Status 404 returned error can't find the container with id b1b577d249df5a7e808bd0279199d930c9682a4f770bf165d8e1b1f5e4814e8e
Nov 03 14:27:16.739095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.738962    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"]
Nov 03 14:27:16.741684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.741654    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:27:16.741763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.741720    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:27:16.741763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:16.741748    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:27:17.244267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:17.242004    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:17.733676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:17.733641    6481 generic.go:356] "Generic (PLEG): container finished" podID="ae8bde37-0b06-46a8-a187-3569ae35dcbc" containerID="5d1447ae74a34ef98e096d9aadb5a4aef07ec33c60cc2319ab5b675c788e821a" exitCode=0
Nov 03 14:27:17.733872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:17.733736    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n" event={"ID":"ae8bde37-0b06-46a8-a187-3569ae35dcbc","Type":"ContainerDied","Data":"5d1447ae74a34ef98e096d9aadb5a4aef07ec33c60cc2319ab5b675c788e821a"}
Nov 03 14:27:17.735796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:17.735773    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" event={"ID":"9953e53b-bb90-4fad-af85-07532f02339f","Type":"ContainerStarted","Data":"5b7ad408b66d19e98c079a0724021827dfe411c902c733baa634224f4ac307b7"}
Nov 03 14:27:17.735924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:17.735817    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" event={"ID":"9953e53b-bb90-4fad-af85-07532f02339f","Type":"ContainerStarted","Data":"b1b577d249df5a7e808bd0279199d930c9682a4f770bf165d8e1b1f5e4814e8e"}
Nov 03 14:27:17.846436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:17.846381    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podStartSLOduration=1.8463669569999999 podStartE2EDuration="1.846366957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:17.84462898 +0000 UTC m=+915.894660164" watchObservedRunningTime="2026-11-03 14:27:17.846366957 +0000 UTC m=+915.896398140"
Nov 03 14:27:18.379119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:18.379078    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:18.380493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:18.380467    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 14:27:18.380593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:18.380519    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 14:27:18.988422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:18.988394    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:19.127636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.126996    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8\" (UniqueName: \"kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8\") pod \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") "
Nov 03 14:27:19.127636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.127108    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca\") pod \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") "
Nov 03 14:27:19.127636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.127294    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca\") pod \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\" (UID: \"ae8bde37-0b06-46a8-a187-3569ae35dcbc\") "
Nov 03 14:27:19.130374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.130349    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:19.131063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.131041    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca" pod "ae8bde37-0b06-46a8-a187-3569ae35dcbc" (UID: "ae8bde37-0b06-46a8-a187-3569ae35dcbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:27:19.131834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.131809    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca" pod "ae8bde37-0b06-46a8-a187-3569ae35dcbc" (UID: "ae8bde37-0b06-46a8-a187-3569ae35dcbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:19.132245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.132211    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8" pod "ae8bde37-0b06-46a8-a187-3569ae35dcbc" (UID: "ae8bde37-0b06-46a8-a187-3569ae35dcbc"). InnerVolumeSpecName "kube-api-access-6czn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:19.241287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.238448    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czn8\" (UniqueName: \"kubernetes.io/projected/ae8bde37-0b06-46a8-a187-3569ae35dcbc-kube-api-access-6czn8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:19.241287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.238481    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:19.241287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.238497    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8bde37-0b06-46a8-a187-3569ae35dcbc-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:19.377788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.377719    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:19.378942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.378917    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 14:27:19.379051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.378963    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 14:27:19.594652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.594621    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:19.783066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.783011    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"
Nov 03 14:27:19.784328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.784298    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n" event={"ID":"ae8bde37-0b06-46a8-a187-3569ae35dcbc","Type":"ContainerDied","Data":"8bba06536dae886347f5cb2b621a0bab33c29341f878d293ca853f93b5119a0e"}
Nov 03 14:27:19.784519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.784499    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bba06536dae886347f5cb2b621a0bab33c29341f878d293ca853f93b5119a0e"
Nov 03 14:27:19.835049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.835023    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"]
Nov 03 14:27:19.835211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.835191    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy" containerID="cri-o://7b4f7a8c3348a04e71767e5a6e1042277f3ba8f8ab33c96815481c425ed36e94" gracePeriod=30
Nov 03 14:27:19.841831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.841492    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"]
Nov 03 14:27:19.841831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.841711    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy" containerID="cri-o://26e2575d19d1f76c33ea99a064d68fc122f1b7b007a158ce94cfccf063841660" gracePeriod=30
Nov 03 14:27:19.846277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.846252    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"]
Nov 03 14:27:19.846434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.846415    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy" containerID="cri-o://cdfc3cb509f189040d1328637d97b437aeab60587d3e808e6a79380ed0099b7e" gracePeriod=30
Nov 03 14:27:19.850677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.850353    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"]
Nov 03 14:27:19.850677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:19.850537    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy" containerID="cri-o://17c0a9b8cfedf0cacc5a22d1225bdaa53cc37343febce1720e60678dbfbdf514" gracePeriod=30
Nov 03 14:27:20.379079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:20.379043    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 14:27:20.379268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:20.379094    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 14:27:21.378516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:21.378479    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 14:27:21.378983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:21.378531    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 14:27:21.926691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:21.926643    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:22.384310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:22.383284    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 14:27:22.384310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:22.383345    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 14:27:23.405255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:23.378882    6481 patch_prober.go:29] interesting pod/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 03 14:27:23.405255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:23.378950    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 03 14:27:23.751256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:23.738308    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:23.806177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:23.805421    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:23.819936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:23.818385    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:23.909377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:23.909322    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:24.033572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.032327    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:24.276254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.276173    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:24.449259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.448258    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:27:24.449259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.448367    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:24.710310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.706144    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:24.761255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.755649    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:24.761255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.756496    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:24.831733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.830611    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:24.836538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.836510    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:27:24.994271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.990599    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:27:24.994271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:24.991011    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:25.302597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.027777    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:27:25.302597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.028509    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:27:25.302597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.038324    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:27:25.302597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.038678    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:27:25.463867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.447453    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:25.463867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.457303    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:25.510255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.495312    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:27:25.510255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.495385    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:25.510255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.495964    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:25.527201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.519930    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:27:25.527201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.520300    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:25.595309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.592104    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:25.917345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:25.886380    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:27:26.110254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.110202    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:27:26.139853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.139818    6481 generic.go:356] "Generic (PLEG): container finished" podID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerID="7b4f7a8c3348a04e71767e5a6e1042277f3ba8f8ab33c96815481c425ed36e94" exitCode=0
Nov 03 14:27:26.140267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.140223    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" event={"ID":"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71","Type":"ContainerDied","Data":"7b4f7a8c3348a04e71767e5a6e1042277f3ba8f8ab33c96815481c425ed36e94"}
Nov 03 14:27:26.155630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.151064    6481 generic.go:356] "Generic (PLEG): container finished" podID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerID="26e2575d19d1f76c33ea99a064d68fc122f1b7b007a158ce94cfccf063841660" exitCode=0
Nov 03 14:27:26.155630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.151163    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" event={"ID":"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a","Type":"ContainerDied","Data":"26e2575d19d1f76c33ea99a064d68fc122f1b7b007a158ce94cfccf063841660"}
Nov 03 14:27:26.161505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.158317    6481 generic.go:356] "Generic (PLEG): container finished" podID="053a3628-1b59-4617-801f-21d29607180b" containerID="17c0a9b8cfedf0cacc5a22d1225bdaa53cc37343febce1720e60678dbfbdf514" exitCode=0
Nov 03 14:27:26.161505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.158487    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" event={"ID":"053a3628-1b59-4617-801f-21d29607180b","Type":"ContainerDied","Data":"17c0a9b8cfedf0cacc5a22d1225bdaa53cc37343febce1720e60678dbfbdf514"}
Nov 03 14:27:26.161505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.158534    6481 scope.go:122] "RemoveContainer" containerID="becef13b733565201b43de0616646848747ca94c2390ebb7f29f56b270d093d4"
Nov 03 14:27:26.165557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.165517    6481 generic.go:356] "Generic (PLEG): container finished" podID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerID="cdfc3cb509f189040d1328637d97b437aeab60587d3e808e6a79380ed0099b7e" exitCode=0
Nov 03 14:27:26.165736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.165711    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" event={"ID":"69b619dc-628e-4fec-b7dc-e8d936dd1a98","Type":"ContainerDied","Data":"cdfc3cb509f189040d1328637d97b437aeab60587d3e808e6a79380ed0099b7e"}
Nov 03 14:27:26.348313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.348289    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:27:26.350637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.350618    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:27:26.354791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.354772    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:27:26.358621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.358600    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:27:26.437967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.437878    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.437967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.437928    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.437967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.437965    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.437988    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438015    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438042    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438075    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438108    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438156    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438184    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.438260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438249    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438313    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438325    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438398    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438405    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438441    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438484    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438519    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438562    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438592    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438626    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.438676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438656    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438688    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438719    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438755    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438792    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438818    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438846    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438887    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438918    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.438975    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439032    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439062    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439102    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439131    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439149    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy\") pod \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\" (UID: \"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439201    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439265    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.439296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439299    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket\") pod \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\" (UID: \"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71\") "
Nov 03 14:27:26.440211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439370    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data\") pod \"053a3628-1b59-4617-801f-21d29607180b\" (UID: \"053a3628-1b59-4617-801f-21d29607180b\") "
Nov 03 14:27:26.440211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439443    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.440211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439472    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo\") pod \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\" (UID: \"69b619dc-628e-4fec-b7dc-e8d936dd1a98\") "
Nov 03 14:27:26.440211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.439724    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440278    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440302    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440317    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440331    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/053a3628-1b59-4617-801f-21d29607180b-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440345    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440357    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.440511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440371    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.440983    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441069    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441105    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441163    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441212    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441325    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.441594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441478    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.442045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441743    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.442045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.441959    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.442329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.442301    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:27:26.442690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.442663    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.442829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.442808    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.443094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.442993    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.443192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.443062    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.443192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.443068    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.443192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.443145    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.444214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444076    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:27:26.444214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444090    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.444214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444129    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "kube-api-access-d6clz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.444214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444166    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "kube-api-access-wvgdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.444214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444191    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "kube-api-access-p2xsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.444616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444348    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:27:26.444616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444424    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "kube-api-access-ggtb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.444616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.444558    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.445154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.445130    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo" pod "053a3628-1b59-4617-801f-21d29607180b" (UID: "053a3628-1b59-4617-801f-21d29607180b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:27:26.445154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.445138    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token" pod "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" (UID: "b76b3d2c-d2e0-4fcb-a06b-867851d1dd71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.445347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.445155    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.445488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.445470    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy" pod "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" (UID: "6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:26.445868 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.445849    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token" pod "69b619dc-628e-4fec-b7dc-e8d936dd1a98" (UID: "69b619dc-628e-4fec-b7dc-e8d936dd1a98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:26.541064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541033    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6clz\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-kube-api-access-d6clz\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541063    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541076    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvgdv\" (UniqueName: \"kubernetes.io/projected/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-kube-api-access-wvgdv\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541088    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541100    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2xsl\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-kube-api-access-p2xsl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541113    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541126    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541139    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541154    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541166    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541178    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541191    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541205    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541219    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541269    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541285    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541297    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541312    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/053a3628-1b59-4617-801f-21d29607180b-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541339    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541353    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggtb5\" (UniqueName: \"kubernetes.io/projected/053a3628-1b59-4617-801f-21d29607180b-kube-api-access-ggtb5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541375    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541389    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541401    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541414    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541426    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541438    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541453    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/053a3628-1b59-4617-801f-21d29607180b-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541465    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b619dc-628e-4fec-b7dc-e8d936dd1a98-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.541568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.541476    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:26.813809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:26.813706    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:27.173804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.173777    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"
Nov 03 14:27:27.174030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.173831    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5" event={"ID":"053a3628-1b59-4617-801f-21d29607180b","Type":"ContainerDied","Data":"98539b3980adc671866587a09040494b3dbb738d4c0dd463058c023550b8baa5"}
Nov 03 14:27:27.174030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.173892    6481 scope.go:122] "RemoveContainer" containerID="17c0a9b8cfedf0cacc5a22d1225bdaa53cc37343febce1720e60678dbfbdf514"
Nov 03 14:27:27.175458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.175439    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"
Nov 03 14:27:27.175584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.175481    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz" event={"ID":"69b619dc-628e-4fec-b7dc-e8d936dd1a98","Type":"ContainerDied","Data":"5ca788a911af56ab746e067f033eb519b99d3da327f7df7338582b3b31503e66"}
Nov 03 14:27:27.177208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.177182    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"
Nov 03 14:27:27.177338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.177247    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s" event={"ID":"b76b3d2c-d2e0-4fcb-a06b-867851d1dd71","Type":"ContainerDied","Data":"65355fa354c9c19de4df761b1049c737173267692d56785882a2523c1af6e28a"}
Nov 03 14:27:27.179442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.179409    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm" event={"ID":"6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a","Type":"ContainerDied","Data":"fd0d36a8a11fc3f988f906b4ceb816a81fed6f3f698d65820c41fc395b5d36d6"}
Nov 03 14:27:27.179442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.179435    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"
Nov 03 14:27:27.195075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.195053    6481 scope.go:122] "RemoveContainer" containerID="cdfc3cb509f189040d1328637d97b437aeab60587d3e808e6a79380ed0099b7e"
Nov 03 14:27:27.204390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.204371    6481 scope.go:122] "RemoveContainer" containerID="7b4f7a8c3348a04e71767e5a6e1042277f3ba8f8ab33c96815481c425ed36e94"
Nov 03 14:27:27.213613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.213596    6481 scope.go:122] "RemoveContainer" containerID="26e2575d19d1f76c33ea99a064d68fc122f1b7b007a158ce94cfccf063841660"
Nov 03 14:27:27.238312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.238287    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"]
Nov 03 14:27:27.250859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.250839    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-xrpdz"]
Nov 03 14:27:27.325747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.325723    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"]
Nov 03 14:27:27.333511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.333487    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-675z5"]
Nov 03 14:27:27.374527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.374503    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"]
Nov 03 14:27:27.380576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.380555    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8jh2s"]
Nov 03 14:27:27.420408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.420382    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"]
Nov 03 14:27:27.427103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:27.427042    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-6d2jm"]
Nov 03 14:27:27.989354 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:27:27.989315    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:27:28.067451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:28.067420    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="053a3628-1b59-4617-801f-21d29607180b" path="/var/lib/kubelet/pods/053a3628-1b59-4617-801f-21d29607180b/volumes"
Nov 03 14:27:28.068612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:28.068573    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" path="/var/lib/kubelet/pods/69b619dc-628e-4fec-b7dc-e8d936dd1a98/volumes"
Nov 03 14:27:28.069464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:28.069438    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" path="/var/lib/kubelet/pods/6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a/volumes"
Nov 03 14:27:28.071241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:28.070220    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" path="/var/lib/kubelet/pods/b76b3d2c-d2e0-4fcb-a06b-867851d1dd71/volumes"
Nov 03 14:27:29.019969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:29.019941    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:29.711435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:29.711406    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"]
Nov 03 14:27:30.119335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119263    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"]
Nov 03 14:27:30.119825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119810    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy"
Nov 03 14:27:30.119871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119826    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy"
Nov 03 14:27:30.119871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119845    6481 cpu_manager.go:414] "containerMap: removing container" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy"
Nov 03 14:27:30.119871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119850    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy"
Nov 03 14:27:30.119871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119859    6481 cpu_manager.go:414] "containerMap: removing container" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy"
Nov 03 14:27:30.119871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119865    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119884    6481 cpu_manager.go:414] "containerMap: removing container" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119888    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119895    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ae8bde37-0b06-46a8-a187-3569ae35dcbc" containerName="revoke-keys"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119900    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8bde37-0b06-46a8-a187-3569ae35dcbc" containerName="revoke-keys"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119917    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.119922    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy"
Nov 03 14:27:30.120012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.120010    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy"
Nov 03 14:27:30.120207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.120019    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b619dc-628e-4fec-b7dc-e8d936dd1a98" containerName="istio-proxy"
Nov 03 14:27:30.120207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.120027    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="053a3628-1b59-4617-801f-21d29607180b" containerName="istio-proxy"
Nov 03 14:27:30.120207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.120032    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8bde37-0b06-46a8-a187-3569ae35dcbc" containerName="revoke-keys"
Nov 03 14:27:30.120207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.120042    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c8a4a1c-82ad-4482-b4a1-7c64c6bea25a" containerName="istio-proxy"
Nov 03 14:27:30.120207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.120050    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b76b3d2c-d2e0-4fcb-a06b-867851d1dd71" containerName="istio-proxy"
Nov 03 14:27:30.124941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.124925    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.145404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.145382    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:27:30.145654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.145632    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:27:30.146915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.146895    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"]
Nov 03 14:27:30.155535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.155515    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:27:30.155648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.155589    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:27:30.285503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.285474    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87c8\" (UniqueName: \"kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.285651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.285513    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.285651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.285604    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.386760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.386689    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87c8\" (UniqueName: \"kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.386760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.386730    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.386949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.386925    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.387580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.387557    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.389056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.389035    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.409151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.409125    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87c8\" (UniqueName: \"kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8\") pod \"maas-api-revoke-sub-26b20c92-ldc5r\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.439119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.439098    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:30.758142 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:30.758111    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62e95e0_62fb_49af_a549_dfbca654ab20.slice/crio-d17e691b3d53abf50c6f684299355f5eccde54a762c975772fc04f7bcd023d45 WatchSource:0}: Error finding container d17e691b3d53abf50c6f684299355f5eccde54a762c975772fc04f7bcd023d45: Status 404 returned error can't find the container with id d17e691b3d53abf50c6f684299355f5eccde54a762c975772fc04f7bcd023d45
Nov 03 14:27:30.759118 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.759093    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"]
Nov 03 14:27:30.857468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:30.857438    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:31.245268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.244187    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r" event={"ID":"e62e95e0-62fb-49af-a549-dfbca654ab20","Type":"ContainerStarted","Data":"f596ff02e35b234d33d77981fab5b337ab79529102eb8c14b3eb5ed07d4e8114"}
Nov 03 14:27:31.245268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.244277    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r" event={"ID":"e62e95e0-62fb-49af-a549-dfbca654ab20","Type":"ContainerStarted","Data":"d17e691b3d53abf50c6f684299355f5eccde54a762c975772fc04f7bcd023d45"}
Nov 03 14:27:31.290547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.290496    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r" podStartSLOduration=1.290484126 podStartE2EDuration="1.290484126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:31.287675556 +0000 UTC m=+929.337706738" watchObservedRunningTime="2026-11-03 14:27:31.290484126 +0000 UTC m=+929.340515308"
Nov 03 14:27:31.869389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.869355    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"]
Nov 03 14:27:31.874214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.874147    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:31.902818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.902784    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:31.906986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.906956    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-b4361f-openshift-default-dockercfg-h8cr5\""
Nov 03 14:27:31.912501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:31.912475    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"]
Nov 03 14:27:32.013246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013197    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013340    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013426    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013529    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqsj5\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013627    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013724    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013811    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013845    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.013874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.013867    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.048733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.048707    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"]
Nov 03 14:27:32.053502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.053477    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.073406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.073377    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-86c2e2\"/\"distinct-w0-86c2e2-kserve-self-signed-certs\""
Nov 03 14:27:32.076439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.076416    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"]
Nov 03 14:27:32.115170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115143    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115185    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115203    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115222    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115342    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115427    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115530    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115594    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqsj5\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115617    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115668    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115676    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115720    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.115954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.115929    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.116197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.116179    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.117461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.117443    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.118100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.118077    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.139519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.139459    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.139643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.139627    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqsj5\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.195881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.195857    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:32.196760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.196741    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:32.217592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.217567    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.217686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.217668    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmw2k\" (UniqueName: \"kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.217737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.217703    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.217904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.217874    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.218019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.217914    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.218019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.217968    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.276687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.274891    6481 generic.go:356] "Generic (PLEG): container finished" podID="e62e95e0-62fb-49af-a549-dfbca654ab20" containerID="f596ff02e35b234d33d77981fab5b337ab79529102eb8c14b3eb5ed07d4e8114" exitCode=0
Nov 03 14:27:32.276687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.275217    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r" event={"ID":"e62e95e0-62fb-49af-a549-dfbca654ab20","Type":"ContainerDied","Data":"f596ff02e35b234d33d77981fab5b337ab79529102eb8c14b3eb5ed07d4e8114"}
Nov 03 14:27:32.319860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.319813    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.319885    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.319918    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.320263    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.320282    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.320652    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.320713    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.320719    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmw2k\" (UniqueName: \"kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.320800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.320784    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.323330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.323304    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.323415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.323343    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.347204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.345513    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmw2k\" (UniqueName: \"kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k\") pod \"distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") " pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.367109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.367064    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:32.524373 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:32.524343    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2378d404_dfd6_420c_8455_a31b8dbb7847.slice/crio-f0a81e78bebd13898e3b18a029a11e19aaf837641dd42990bb696fba34f8f397 WatchSource:0}: Error finding container f0a81e78bebd13898e3b18a029a11e19aaf837641dd42990bb696fba34f8f397: Status 404 returned error can't find the container with id f0a81e78bebd13898e3b18a029a11e19aaf837641dd42990bb696fba34f8f397
Nov 03 14:27:32.525017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.524990    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"]
Nov 03 14:27:32.527518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.527483    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:27:32.527599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.527562    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:27:32.527672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.527602    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:27:32.687441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:32.687412    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"]
Nov 03 14:27:32.688048 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:32.688022    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5f1f415_bdd9_4a58_82ca_63ac582267f2.slice/crio-7aac73a6e04aca3cbdd0c257897a0384bef217fed81e14dfbf2c8a626d7358bd WatchSource:0}: Error finding container 7aac73a6e04aca3cbdd0c257897a0384bef217fed81e14dfbf2c8a626d7358bd: Status 404 returned error can't find the container with id 7aac73a6e04aca3cbdd0c257897a0384bef217fed81e14dfbf2c8a626d7358bd
Nov 03 14:27:33.284887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.284853    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" event={"ID":"d5f1f415-bdd9-4a58-82ca-63ac582267f2","Type":"ContainerStarted","Data":"fa8f4db23ce63c949329b19dff3919b39ed77bf1974117d7f281911ba0177c41"}
Nov 03 14:27:33.285289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.284896    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" event={"ID":"d5f1f415-bdd9-4a58-82ca-63ac582267f2","Type":"ContainerStarted","Data":"7aac73a6e04aca3cbdd0c257897a0384bef217fed81e14dfbf2c8a626d7358bd"}
Nov 03 14:27:33.286607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.286585    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" event={"ID":"2378d404-dfd6-420c-8455-a31b8dbb7847","Type":"ContainerStarted","Data":"b7e920cfd75f6a224775f3a114b906acb17c49ae1979b3a12fde6c5c4273b4b2"}
Nov 03 14:27:33.286734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.286625    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" event={"ID":"2378d404-dfd6-420c-8455-a31b8dbb7847","Type":"ContainerStarted","Data":"f0a81e78bebd13898e3b18a029a11e19aaf837641dd42990bb696fba34f8f397"}
Nov 03 14:27:33.350002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.349942    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podStartSLOduration=1.349927538 podStartE2EDuration="1.349927538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:33.348760173 +0000 UTC m=+931.398791356" watchObservedRunningTime="2026-11-03 14:27:33.349927538 +0000 UTC m=+931.399958738"
Nov 03 14:27:33.368012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.367978    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:33.430301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.417131    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podStartSLOduration=2.417109778 podStartE2EDuration="2.417109778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:33.41577835 +0000 UTC m=+931.465809535" watchObservedRunningTime="2026-11-03 14:27:33.417109778 +0000 UTC m=+931.467140985"
Nov 03 14:27:33.470570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.470544    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:33.639887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.639806    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca\") pod \"e62e95e0-62fb-49af-a549-dfbca654ab20\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") "
Nov 03 14:27:33.640219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.640201    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca\") pod \"e62e95e0-62fb-49af-a549-dfbca654ab20\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") "
Nov 03 14:27:33.640330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.640317    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8\" (UniqueName: \"kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8\") pod \"e62e95e0-62fb-49af-a549-dfbca654ab20\" (UID: \"e62e95e0-62fb-49af-a549-dfbca654ab20\") "
Nov 03 14:27:33.640631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.640604    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca" pod "e62e95e0-62fb-49af-a549-dfbca654ab20" (UID: "e62e95e0-62fb-49af-a549-dfbca654ab20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:33.641049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.641018    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:33.642090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.642058    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca" pod "e62e95e0-62fb-49af-a549-dfbca654ab20" (UID: "e62e95e0-62fb-49af-a549-dfbca654ab20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:27:33.642437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.642415    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8" pod "e62e95e0-62fb-49af-a549-dfbca654ab20" (UID: "e62e95e0-62fb-49af-a549-dfbca654ab20"). InnerVolumeSpecName "kube-api-access-z87c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:33.742821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.742785    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87c8\" (UniqueName: \"kubernetes.io/projected/e62e95e0-62fb-49af-a549-dfbca654ab20-kube-api-access-z87c8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:33.742821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:33.742819    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62e95e0-62fb-49af-a549-dfbca654ab20-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:34.130077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.130046    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:34.197012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.196978    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:34.198714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.198673    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:34.198840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.198728    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:34.327694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.327657    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"
Nov 03 14:27:34.327694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.327675    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r" event={"ID":"e62e95e0-62fb-49af-a549-dfbca654ab20","Type":"ContainerDied","Data":"d17e691b3d53abf50c6f684299355f5eccde54a762c975772fc04f7bcd023d45"}
Nov 03 14:27:34.328177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.327779    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d17e691b3d53abf50c6f684299355f5eccde54a762c975772fc04f7bcd023d45"
Nov 03 14:27:34.593444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.593410    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:27:34.721704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:34.721662    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w" podUID="9878d82f-ee00-4f1c-8a07-035deb8750ad" containerName="payload-processing" containerID="cri-o://6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7" gracePeriod=25
Nov 03 14:27:35.047831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.047807    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:27:35.161108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161083    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume\") pod \"9878d82f-ee00-4f1c-8a07-035deb8750ad\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") "
Nov 03 14:27:35.161269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161189    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials\") pod \"9878d82f-ee00-4f1c-8a07-035deb8750ad\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") "
Nov 03 14:27:35.161368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161350    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr\") pod \"9878d82f-ee00-4f1c-8a07-035deb8750ad\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") "
Nov 03 14:27:35.161408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161390    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume\") pod \"9878d82f-ee00-4f1c-8a07-035deb8750ad\" (UID: \"9878d82f-ee00-4f1c-8a07-035deb8750ad\") "
Nov 03 14:27:35.161546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161519    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume" pod "9878d82f-ee00-4f1c-8a07-035deb8750ad" (UID: "9878d82f-ee00-4f1c-8a07-035deb8750ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:35.161864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161829    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume" pod "9878d82f-ee00-4f1c-8a07-035deb8750ad" (UID: "9878d82f-ee00-4f1c-8a07-035deb8750ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:35.161978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161953    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:35.161978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.161974    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9878d82f-ee00-4f1c-8a07-035deb8750ad-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:35.163569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.163547    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr" pod "9878d82f-ee00-4f1c-8a07-035deb8750ad" (UID: "9878d82f-ee00-4f1c-8a07-035deb8750ad"). InnerVolumeSpecName "kube-api-access-4hdpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:35.163569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.163559    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials" pod "9878d82f-ee00-4f1c-8a07-035deb8750ad" (UID: "9878d82f-ee00-4f1c-8a07-035deb8750ad"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:35.197809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.197785    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:35.197912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.197839    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:35.263761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.263732    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:35.263761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.263758    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hdpr\" (UniqueName: \"kubernetes.io/projected/9878d82f-ee00-4f1c-8a07-035deb8750ad-kube-api-access-4hdpr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:35.348252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.348052    6481 generic.go:356] "Generic (PLEG): container finished" podID="9878d82f-ee00-4f1c-8a07-035deb8750ad" containerID="6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7" exitCode=0
Nov 03 14:27:35.351254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.349066    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"
Nov 03 14:27:35.355248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.352668    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w" event={"ID":"9878d82f-ee00-4f1c-8a07-035deb8750ad","Type":"ContainerDied","Data":"6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7"}
Nov 03 14:27:35.355248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.352746    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w" event={"ID":"9878d82f-ee00-4f1c-8a07-035deb8750ad","Type":"ContainerDied","Data":"c0170fc3bc7ff8fe7b21714a58cecbf001cd195fc88820a41a4d99c5af1093e5"}
Nov 03 14:27:35.355248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.352778    6481 scope.go:122] "RemoveContainer" containerID="6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7"
Nov 03 14:27:35.456418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.444421    6481 scope.go:122] "RemoveContainer" containerID="6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7"
Nov 03 14:27:35.456418 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:27:35.451427    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7\": container with ID starting with 6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7 not found: ID does not exist" containerID="6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7"
Nov 03 14:27:35.456418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.451473    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7"} err="failed to get container status \"6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7\": rpc error: code = NotFound desc = could not find container \"6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7\": container with ID starting with 6cb21f7f041585a4093ecdc1842af12e67512b2d80dd07c302c872b4c3e179b7 not found: ID does not exist"
Nov 03 14:27:35.477337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.477307    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"]
Nov 03 14:27:35.504272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:35.502697    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-7cc5dp42w"]
Nov 03 14:27:36.114265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.110968    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9878d82f-ee00-4f1c-8a07-035deb8750ad" path="/var/lib/kubelet/pods/9878d82f-ee00-4f1c-8a07-035deb8750ad/volumes"
Nov 03 14:27:36.198326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.197628    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:36.198326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.197707    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:36.426268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.420323    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": context deadline exceeded"
Nov 03 14:27:36.530266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.527322    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:36.530266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.528630    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:27:36.541377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.541320    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:36.617265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.613319    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:36.780322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.776169    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:36.894266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:36.891321    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:37.204266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.200671    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:37.204266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.200745    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:37.262254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.261193    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:37.539313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.538730    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:37.539313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.538794    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:27:37.539313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.539157    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:37.539313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.539217    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:27:37.591015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.590340    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:37.591015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.590530    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:27:37.616400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.614345    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:37.616400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:37.614736    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:38.106417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.041816    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:27:38.106417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.042528    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:27:38.445393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.444253    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:38.445393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.444331    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:38.555341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.555302    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:27:38.556947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.556918    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:27:38.644627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.644577    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:38.672569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:38.671415    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:39.044121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.044080    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:39.049057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.049028    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:39.184684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.184655    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:27:39.231466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.231421    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:39.231756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.231716    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:39.394302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.385538    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:39.461419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.460581    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:39.492254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.489658    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:27:39.498248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:39.497709    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:27:40.197774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:40.197741    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 03 14:27:40.198201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:40.197793    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 03 14:27:41.109259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:41.109193    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:41.201813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:41.201783    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:41.202293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:41.202147    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:41.202948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:41.202921    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:27:42.367459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:42.367422    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:42.379426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:42.379403    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:42.584836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:42.584806    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:43.268279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:43.264999    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:44.409298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.409254    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"]
Nov 03 14:27:44.409941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.409919    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9878d82f-ee00-4f1c-8a07-035deb8750ad" containerName="payload-processing"
Nov 03 14:27:44.410028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.409942    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9878d82f-ee00-4f1c-8a07-035deb8750ad" containerName="payload-processing"
Nov 03 14:27:44.410028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.409973    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e62e95e0-62fb-49af-a549-dfbca654ab20" containerName="revoke-keys"
Nov 03 14:27:44.410028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.409980    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62e95e0-62fb-49af-a549-dfbca654ab20" containerName="revoke-keys"
Nov 03 14:27:44.410130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.410092    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9878d82f-ee00-4f1c-8a07-035deb8750ad" containerName="payload-processing"
Nov 03 14:27:44.410130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.410103    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62e95e0-62fb-49af-a549-dfbca654ab20" containerName="revoke-keys"
Nov 03 14:27:44.424164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.424138    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.433654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.433628    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:44.436022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.435999    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"]
Nov 03 14:27:44.440005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.439985    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:27:44.443544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.443520    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:27:44.443630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.443574    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:27:44.443760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.443742    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:27:44.505354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.505317    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.505524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.505377    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcl4\" (UniqueName: \"kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.505586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.505542    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.607644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.607597    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.607819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.607662    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcl4\" (UniqueName: \"kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.607819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.607754    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.608284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.608250    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.610136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.610118    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.629893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.629867    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcl4\" (UniqueName: \"kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4\") pod \"maas-api-revoke-sub-57e85451-77mq4\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.644151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.644131    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:44.740525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.740432    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:44.876947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:44.876746    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:45.206792 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:45.206749    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eebfe6_45f6_4ca2_9e56_c2043bb54979.slice/crio-18d4edab5949bb0136fdae13c04e0979a46c73c4922854ca762527005a93922e WatchSource:0}: Error finding container 18d4edab5949bb0136fdae13c04e0979a46c73c4922854ca762527005a93922e: Status 404 returned error can't find the container with id 18d4edab5949bb0136fdae13c04e0979a46c73c4922854ca762527005a93922e
Nov 03 14:27:45.210392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:45.210368    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"]
Nov 03 14:27:45.606183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:45.606139    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4" event={"ID":"17eebfe6-45f6-4ca2-9e56-c2043bb54979","Type":"ContainerStarted","Data":"edf727f200dc5768118de55d59386c310e7abb1c5ad53a9a5161e817a8a1b709"}
Nov 03 14:27:45.606734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:45.606237    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4" event={"ID":"17eebfe6-45f6-4ca2-9e56-c2043bb54979","Type":"ContainerStarted","Data":"18d4edab5949bb0136fdae13c04e0979a46c73c4922854ca762527005a93922e"}
Nov 03 14:27:45.654244 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:45.654188    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4" podStartSLOduration=1.654174023 podStartE2EDuration="1.654174023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:45.652216426 +0000 UTC m=+943.702247610" watchObservedRunningTime="2026-11-03 14:27:45.654174023 +0000 UTC m=+943.704205206"
Nov 03 14:27:45.747400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:45.747116    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:45.958894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:45.958838    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:46.070451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.070424    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"]
Nov 03 14:27:46.076125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.076103    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.081698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.081668    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"]
Nov 03 14:27:46.081904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.081878    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy" containerID="cri-o://5b7ad408b66d19e98c079a0724021827dfe411c902c733baa634224f4ac307b7" gracePeriod=30
Nov 03 14:27:46.085747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.085722    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"]
Nov 03 14:27:46.085976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.085931    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy" containerID="cri-o://be29bf6fb765d20445fea83da87ae2581f19b6d2641a9cada1d607967bb7aaa4" gracePeriod=30
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.094722    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-b28d34176fc3\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.094778    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-c88df00e610c\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.095040    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.095078    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.095186    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.095198    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.095205    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\""
Nov 03 14:27:46.099259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.095287    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\""
Nov 03 14:27:46.099951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.099932    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"]
Nov 03 14:27:46.241335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.228129    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:46.260338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.260305    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.260654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.260620    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cr2\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.260919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.260891    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.261159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.261132    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.362890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.362793    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.363075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.363022    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.363075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.363056    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cr2\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.363216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.363200    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.363562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.363534    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.363762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.363736    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.366021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.365991    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.385791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.385769    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cr2\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.394010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.393983    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:46.615827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.615739    6481 generic.go:356] "Generic (PLEG): container finished" podID="17eebfe6-45f6-4ca2-9e56-c2043bb54979" containerID="edf727f200dc5768118de55d59386c310e7abb1c5ad53a9a5161e817a8a1b709" exitCode=0
Nov 03 14:27:46.616281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.615827    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4" event={"ID":"17eebfe6-45f6-4ca2-9e56-c2043bb54979","Type":"ContainerDied","Data":"edf727f200dc5768118de55d59386c310e7abb1c5ad53a9a5161e817a8a1b709"}
Nov 03 14:27:46.705393 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:46.705363    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b1c21a_bb43_48c8_9add_376c566a87c6.slice/crio-fc207a184243307527fa4c198ad3fe8589e6f7a64bfeb28a0bef297270308cdd WatchSource:0}: Error finding container fc207a184243307527fa4c198ad3fe8589e6f7a64bfeb28a0bef297270308cdd: Status 404 returned error can't find the container with id fc207a184243307527fa4c198ad3fe8589e6f7a64bfeb28a0bef297270308cdd
Nov 03 14:27:46.706372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.706349    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"]
Nov 03 14:27:46.985268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:46.985239    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:47.258555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.258464    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:47.625320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.625207    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9" event={"ID":"f5b1c21a-bb43-48c8-9add-376c566a87c6","Type":"ContainerStarted","Data":"50e24e964eb19e796e23ef366fcf6031f3165be2ae8abdee3b423931f49e8757"}
Nov 03 14:27:47.625320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.625294    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9" event={"ID":"f5b1c21a-bb43-48c8-9add-376c566a87c6","Type":"ContainerStarted","Data":"fc207a184243307527fa4c198ad3fe8589e6f7a64bfeb28a0bef297270308cdd"}
Nov 03 14:27:47.625812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.625535    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:47.855302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.855279    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:47.901047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.900952    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca\") pod \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") "
Nov 03 14:27:47.901846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.901084    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4\" (UniqueName: \"kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4\") pod \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") "
Nov 03 14:27:47.901846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.901125    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca\") pod \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\" (UID: \"17eebfe6-45f6-4ca2-9e56-c2043bb54979\") "
Nov 03 14:27:47.902614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.902022    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca" pod "17eebfe6-45f6-4ca2-9e56-c2043bb54979" (UID: "17eebfe6-45f6-4ca2-9e56-c2043bb54979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:47.903435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.903383    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9" podStartSLOduration=1.903365344 podStartE2EDuration="1.903365344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:47.696543958 +0000 UTC m=+945.746575141" watchObservedRunningTime="2026-11-03 14:27:47.903365344 +0000 UTC m=+945.953396530"
Nov 03 14:27:47.903976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.903939    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca" pod "17eebfe6-45f6-4ca2-9e56-c2043bb54979" (UID: "17eebfe6-45f6-4ca2-9e56-c2043bb54979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:27:47.904480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:47.904448    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4" pod "17eebfe6-45f6-4ca2-9e56-c2043bb54979" (UID: "17eebfe6-45f6-4ca2-9e56-c2043bb54979"). InnerVolumeSpecName "kube-api-access-6pcl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:48.002979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.002945    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:48.002979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.002974    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pcl4\" (UniqueName: \"kubernetes.io/projected/17eebfe6-45f6-4ca2-9e56-c2043bb54979-kube-api-access-6pcl4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:48.002979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.002985    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17eebfe6-45f6-4ca2-9e56-c2043bb54979-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:48.241039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.241008    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:48.543654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.543585    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"]
Nov 03 14:27:48.544162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.544149    6481 cpu_manager.go:414] "containerMap: removing container" podUID="17eebfe6-45f6-4ca2-9e56-c2043bb54979" containerName="revoke-keys"
Nov 03 14:27:48.544210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.544164    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17eebfe6-45f6-4ca2-9e56-c2043bb54979" containerName="revoke-keys"
Nov 03 14:27:48.544321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.544311    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="17eebfe6-45f6-4ca2-9e56-c2043bb54979" containerName="revoke-keys"
Nov 03 14:27:48.547687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.547664    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.570025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.570000    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"]
Nov 03 14:27:48.611245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.611209    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.611390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.611320    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc865\" (UniqueName: \"kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.611591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.611562    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.634204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.634185    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"
Nov 03 14:27:48.634535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.634217    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4" event={"ID":"17eebfe6-45f6-4ca2-9e56-c2043bb54979","Type":"ContainerDied","Data":"18d4edab5949bb0136fdae13c04e0979a46c73c4922854ca762527005a93922e"}
Nov 03 14:27:48.634535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.634282    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18d4edab5949bb0136fdae13c04e0979a46c73c4922854ca762527005a93922e"
Nov 03 14:27:48.712579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.712552    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.712733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.712622    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc865\" (UniqueName: \"kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.712800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.712782    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.713298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.713271    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.715040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.715022    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.736411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.736381    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc865\" (UniqueName: \"kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865\") pod \"maas-api-revoke-sub-ea5f5e2d-zfft8\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.862064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.861985    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:48.917448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:48.917415    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:27:49.173948 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:49.173915    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591e01e5_7a96_4a7d_80ce_a51acafd6f2a.slice/crio-6eafc3ea6ff4a9d266bbced545c1e9d99e922096f696ce3e3b62c07e0e075cd1 WatchSource:0}: Error finding container 6eafc3ea6ff4a9d266bbced545c1e9d99e922096f696ce3e3b62c07e0e075cd1: Status 404 returned error can't find the container with id 6eafc3ea6ff4a9d266bbced545c1e9d99e922096f696ce3e3b62c07e0e075cd1
Nov 03 14:27:49.174804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:49.174726    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"]
Nov 03 14:27:49.683088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:49.682994    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8" event={"ID":"591e01e5-7a96-4a7d-80ce-a51acafd6f2a","Type":"ContainerStarted","Data":"6eafc3ea6ff4a9d266bbced545c1e9d99e922096f696ce3e3b62c07e0e075cd1"}
Nov 03 14:27:50.223306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:50.117360    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:50.750312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:50.734122    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:50.858064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:50.858020    6481 generic.go:356] "Generic (PLEG): container finished" podID="591e01e5-7a96-4a7d-80ce-a51acafd6f2a" containerID="e2e44ebd5d528c6f98a4359f58c59c6ecd9d58ab8f91c3e5c0f84803fc4d2103" exitCode=0
Nov 03 14:27:50.876949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:50.876349    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8" event={"ID":"591e01e5-7a96-4a7d-80ce-a51acafd6f2a","Type":"ContainerDied","Data":"e2e44ebd5d528c6f98a4359f58c59c6ecd9d58ab8f91c3e5c0f84803fc4d2103"}
Nov 03 14:27:51.135401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.135358    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:51.516527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.515333    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:27:51.555799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.555066    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:51.583253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.582377    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:27:51.597378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.597335    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:51.730403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.730354    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:51.920047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:51.919945    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:52.310706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.233401    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:27:52.386296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.386117    6481 generic.go:356] "Generic (PLEG): container finished" podID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerID="be29bf6fb765d20445fea83da87ae2581f19b6d2641a9cada1d607967bb7aaa4" exitCode=0
Nov 03 14:27:52.604513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.604378    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Nov 03 14:27:52.631261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.626176    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:52.656275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.652168    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:52.664274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.656847    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:52.795121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.720089    6481 generic.go:356] "Generic (PLEG): container finished" podID="9953e53b-bb90-4fad-af85-07532f02339f" containerID="5b7ad408b66d19e98c079a0724021827dfe411c902c733baa634224f4ac307b7" exitCode=0
Nov 03 14:27:52.795121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:52.747618    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:27:53.463317 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:27:53.462584    6481 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.392s"
Nov 03 14:27:53.463317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.462760    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" event={"ID":"d8d2777d-f0f8-4f2e-811e-47151be12b1d","Type":"ContainerDied","Data":"be29bf6fb765d20445fea83da87ae2581f19b6d2641a9cada1d607967bb7aaa4"}
Nov 03 14:27:53.520305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.466298    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" event={"ID":"9953e53b-bb90-4fad-af85-07532f02339f","Type":"ContainerDied","Data":"5b7ad408b66d19e98c079a0724021827dfe411c902c733baa634224f4ac307b7"}
Nov 03 14:27:53.520305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.466448    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:27:53.520305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.466521    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:53.556568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.549993    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:27:53.556669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.556568    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:27:53.556733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.556695    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:53.557000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.556781    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:53.564464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.564429    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:53.818990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.771605    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:27:53.818990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.816965    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:27:53.884698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.884671    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:27:53.900541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.900516    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:27:53.900792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.900780    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:27:53.900910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.900901    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:27:53.920257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.914392    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:53.966540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.952173    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8" event={"ID":"9953e53b-bb90-4fad-af85-07532f02339f","Type":"ContainerDied","Data":"b1b577d249df5a7e808bd0279199d930c9682a4f770bf165d8e1b1f5e4814e8e"}
Nov 03 14:27:53.966540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.952266    6481 scope.go:122] "RemoveContainer" containerID="5b7ad408b66d19e98c079a0724021827dfe411c902c733baa634224f4ac307b7"
Nov 03 14:27:53.966540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.952904    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:53.967972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.967928    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.968246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.968195    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.972508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.972459    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.972773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.972748    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.973102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.973083    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.973307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.973289    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.973633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.973616    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.973832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.973810    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.973963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.973950    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data\") pod \"9953e53b-bb90-4fad-af85-07532f02339f\" (UID: \"9953e53b-bb90-4fad-af85-07532f02339f\") "
Nov 03 14:27:53.980412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.980366    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:53.981247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.980874    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:53.982617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.982597    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:27:53.993001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.987913    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9953e53b-bb90-4fad-af85-07532f02339f-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.001296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:53.989026    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.001471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.001341    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.007974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.001360    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6" event={"ID":"d8d2777d-f0f8-4f2e-811e-47151be12b1d","Type":"ContainerDied","Data":"8b408f0248015ea44a3bbe63741d291fff6d3bfbb666342d83cfd05c9fcdea3e"}
Nov 03 14:27:54.007974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.006353    6481 scope.go:122] "RemoveContainer" containerID="be29bf6fb765d20445fea83da87ae2581f19b6d2641a9cada1d607967bb7aaa4"
Nov 03 14:27:54.007974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.003736    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.011356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.011335    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.013045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.013016    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:27:54.013525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.013420    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.013525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.013496    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "kube-api-access-mpmct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:54.014359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.014297    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token" pod "9953e53b-bb90-4fad-af85-07532f02339f" (UID: "9953e53b-bb90-4fad-af85-07532f02339f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:54.057903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.057871    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:54.106599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106568    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.106769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106625    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.106769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106649    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.106769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106702    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.106769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106739    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.106975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106912    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.106975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.106964    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.107072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.107012    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.107124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.107101    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy\") pod \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\" (UID: \"d8d2777d-f0f8-4f2e-811e-47151be12b1d\") "
Nov 03 14:27:54.107375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.107336    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.107937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.107907    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.108055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.107969    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.108517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.108222    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:54.108681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.108647    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.110622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.110599    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:54.110861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.110838    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:27:54.111460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111436    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9953e53b-bb90-4fad-af85-07532f02339f-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111465    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpmct\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-kube-api-access-mpmct\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111484    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111501    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111516    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111528    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111542    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9953e53b-bb90-4fad-af85-07532f02339f-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111555    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111569    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111584    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111599    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111615    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.111636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111629    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.112165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.111659    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9953e53b-bb90-4fad-af85-07532f02339f-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.112302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.112276    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "kube-api-access-24x8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:54.112389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.112354    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy" pod "d8d2777d-f0f8-4f2e-811e-47151be12b1d" (UID: "d8d2777d-f0f8-4f2e-811e-47151be12b1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:27:54.213308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.213271    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865\" (UniqueName: \"kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865\") pod \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") "
Nov 03 14:27:54.213515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.213331    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca\") pod \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") "
Nov 03 14:27:54.213515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.213420    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca\") pod \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\" (UID: \"591e01e5-7a96-4a7d-80ce-a51acafd6f2a\") "
Nov 03 14:27:54.214381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.214042    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8d2777d-f0f8-4f2e-811e-47151be12b1d-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.214381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.214066    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x8n\" (UniqueName: \"kubernetes.io/projected/d8d2777d-f0f8-4f2e-811e-47151be12b1d-kube-api-access-24x8n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.216542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.216482    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca" pod "591e01e5-7a96-4a7d-80ce-a51acafd6f2a" (UID: "591e01e5-7a96-4a7d-80ce-a51acafd6f2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:54.216787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.216732    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca" pod "591e01e5-7a96-4a7d-80ce-a51acafd6f2a" (UID: "591e01e5-7a96-4a7d-80ce-a51acafd6f2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:27:54.219150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.219127    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865" pod "591e01e5-7a96-4a7d-80ce-a51acafd6f2a" (UID: "591e01e5-7a96-4a7d-80ce-a51acafd6f2a"). InnerVolumeSpecName "kube-api-access-xc865". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:54.253782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.253756    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"]
Nov 03 14:27:54.315176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.315139    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc865\" (UniqueName: \"kubernetes.io/projected/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-kube-api-access-xc865\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.315433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.315413    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.315544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.315531    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/591e01e5-7a96-4a7d-80ce-a51acafd6f2a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:54.460432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.460390    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:54.676947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:54.676920    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.020006    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"]
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.020936    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.020955    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.020989    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.020997    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.021050    6481 cpu_manager.go:414] "containerMap: removing container" podUID="591e01e5-7a96-4a7d-80ce-a51acafd6f2a" containerName="revoke-keys"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.021059    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="591e01e5-7a96-4a7d-80ce-a51acafd6f2a" containerName="revoke-keys"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.021202    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9953e53b-bb90-4fad-af85-07532f02339f" containerName="istio-proxy"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.021239    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="591e01e5-7a96-4a7d-80ce-a51acafd6f2a" containerName="revoke-keys"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.021251    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" containerName="istio-proxy"
Nov 03 14:27:55.031314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.025208    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"
Nov 03 14:27:55.039090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.032265    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"
Nov 03 14:27:55.041068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.040310    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"
Nov 03 14:27:55.057363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.057333    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"]
Nov 03 14:27:55.057491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.057377    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8" event={"ID":"591e01e5-7a96-4a7d-80ce-a51acafd6f2a","Type":"ContainerDied","Data":"6eafc3ea6ff4a9d266bbced545c1e9d99e922096f696ce3e3b62c07e0e075cd1"}
Nov 03 14:27:55.057491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.057408    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eafc3ea6ff4a9d266bbced545c1e9d99e922096f696ce3e3b62c07e0e075cd1"
Nov 03 14:27:55.057615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.057512    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.074822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.074802    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:27:55.075025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.075012    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:27:55.075092 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.075067    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:27:55.075168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.075147    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:27:55.090245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.090215    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"]
Nov 03 14:27:55.099021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.098997    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-lsdw8"]
Nov 03 14:27:55.176657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.176625    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"]
Nov 03 14:27:55.184569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.184544    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5b97f1-openshift-default-7cff8c8879-cwgt6"]
Nov 03 14:27:55.231991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.231965    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.232190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.232175    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.232436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.232407    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjsfn\" (UniqueName: \"kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.333446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.333364    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjsfn\" (UniqueName: \"kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.333590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.333458    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.333590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.333555    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.334210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.334186    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.335923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.335901    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.355866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.355840    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjsfn\" (UniqueName: \"kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn\") pod \"maas-api-revoke-sub-785623f9-4shjh\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.372101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.372082    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:55.677305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.677274    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"]
Nov 03 14:27:55.677624 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:55.677456    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99be379e_2cdc_4f36_8413_75bc5f3e788e.slice/crio-5aec6f2fc33273cd063b597057c0ff478e82c4c48b53e2d2478479aef109b829 WatchSource:0}: Error finding container 5aec6f2fc33273cd063b597057c0ff478e82c4c48b53e2d2478479aef109b829: Status 404 returned error can't find the container with id 5aec6f2fc33273cd063b597057c0ff478e82c4c48b53e2d2478479aef109b829
Nov 03 14:27:55.783142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:55.783117    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:56.048550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:56.048513    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh" event={"ID":"99be379e-2cdc-4f36-8413-75bc5f3e788e","Type":"ContainerStarted","Data":"e0135342218f9ce6e271b08828617b2ee0fe8e405910e9c22e73784a2a2a8543"}
Nov 03 14:27:56.048691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:56.048581    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh" event={"ID":"99be379e-2cdc-4f36-8413-75bc5f3e788e","Type":"ContainerStarted","Data":"5aec6f2fc33273cd063b597057c0ff478e82c4c48b53e2d2478479aef109b829"}
Nov 03 14:27:56.068478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:56.068452    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9953e53b-bb90-4fad-af85-07532f02339f" path="/var/lib/kubelet/pods/9953e53b-bb90-4fad-af85-07532f02339f/volumes"
Nov 03 14:27:56.068909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:56.068895    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8d2777d-f0f8-4f2e-811e-47151be12b1d" path="/var/lib/kubelet/pods/d8d2777d-f0f8-4f2e-811e-47151be12b1d/volumes"
Nov 03 14:27:56.092773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:56.092730    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh" podStartSLOduration=2.092717186 podStartE2EDuration="2.092717186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:56.090175576 +0000 UTC m=+954.140206763" watchObservedRunningTime="2026-11-03 14:27:56.092717186 +0000 UTC m=+954.142748418"
Nov 03 14:27:56.203048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:56.203009    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:57.057635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:57.057603    6481 generic.go:356] "Generic (PLEG): container finished" podID="99be379e-2cdc-4f36-8413-75bc5f3e788e" containerID="e0135342218f9ce6e271b08828617b2ee0fe8e405910e9c22e73784a2a2a8543" exitCode=0
Nov 03 14:27:57.058039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:57.057670    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh" event={"ID":"99be379e-2cdc-4f36-8413-75bc5f3e788e","Type":"ContainerDied","Data":"e0135342218f9ce6e271b08828617b2ee0fe8e405910e9c22e73784a2a2a8543"}
Nov 03 14:27:57.627483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:57.627449    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:57.985724 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:27:57.985681    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:27:58.192202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.192176    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:27:58.223351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.223325    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:58.280147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.280087    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca\") pod \"99be379e-2cdc-4f36-8413-75bc5f3e788e\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") "
Nov 03 14:27:58.280298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.280285    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn\" (UniqueName: \"kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn\") pod \"99be379e-2cdc-4f36-8413-75bc5f3e788e\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") "
Nov 03 14:27:58.280340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.280333    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca\") pod \"99be379e-2cdc-4f36-8413-75bc5f3e788e\" (UID: \"99be379e-2cdc-4f36-8413-75bc5f3e788e\") "
Nov 03 14:27:58.280792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.280762    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca" pod "99be379e-2cdc-4f36-8413-75bc5f3e788e" (UID: "99be379e-2cdc-4f36-8413-75bc5f3e788e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:27:58.282208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.282182    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca" pod "99be379e-2cdc-4f36-8413-75bc5f3e788e" (UID: "99be379e-2cdc-4f36-8413-75bc5f3e788e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:27:58.282552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.282528    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn" pod "99be379e-2cdc-4f36-8413-75bc5f3e788e" (UID: "99be379e-2cdc-4f36-8413-75bc5f3e788e"). InnerVolumeSpecName "kube-api-access-gjsfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:27:58.382619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.382583    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjsfn\" (UniqueName: \"kubernetes.io/projected/99be379e-2cdc-4f36-8413-75bc5f3e788e-kube-api-access-gjsfn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:58.382619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.382611    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:58.382619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.382623    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99be379e-2cdc-4f36-8413-75bc5f3e788e-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:27:58.637777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.637710    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:27:58.887077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.887045    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"]
Nov 03 14:27:58.887726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.887710    6481 cpu_manager.go:414] "containerMap: removing container" podUID="99be379e-2cdc-4f36-8413-75bc5f3e788e" containerName="revoke-keys"
Nov 03 14:27:58.887812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.887728    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99be379e-2cdc-4f36-8413-75bc5f3e788e" containerName="revoke-keys"
Nov 03 14:27:58.887859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.887842    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="99be379e-2cdc-4f36-8413-75bc5f3e788e" containerName="revoke-keys"
Nov 03 14:27:58.892020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.892002    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:58.917576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.917551    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"]
Nov 03 14:27:58.990537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.990506    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:58.990674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.990558    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w87h\" (UniqueName: \"kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:58.990717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:58.990677    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.075797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.075765    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"
Nov 03 14:27:59.075948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.075806    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh" event={"ID":"99be379e-2cdc-4f36-8413-75bc5f3e788e","Type":"ContainerDied","Data":"5aec6f2fc33273cd063b597057c0ff478e82c4c48b53e2d2478479aef109b829"}
Nov 03 14:27:59.075948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.075846    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aec6f2fc33273cd063b597057c0ff478e82c4c48b53e2d2478479aef109b829"
Nov 03 14:27:59.091957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.091933    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.092111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.092095    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.092162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.092147    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w87h\" (UniqueName: \"kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.092852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.092832    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.094424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.094401    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.115440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.115412    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w87h\" (UniqueName: \"kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h\") pod \"maas-api-revoke-sub-c7eab127-d8vmb\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.208129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.208105    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:27:59.254953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.254914    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerName="payload-pre-processing" containerID="cri-o://e8da05c80a84a07f5bd00f250fe09b6ae1427d84467f677637f8106df66453c2" gracePeriod=25
Nov 03 14:27:59.512142 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:27:59.512114    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea037f26_764f_4aee_818c_698f952bcad9.slice/crio-1b32a92962578ae86763440a512fc4c7f5de4f4c3c225024fb0807d1125dca8d WatchSource:0}: Error finding container 1b32a92962578ae86763440a512fc4c7f5de4f4c3c225024fb0807d1125dca8d: Status 404 returned error can't find the container with id 1b32a92962578ae86763440a512fc4c7f5de4f4c3c225024fb0807d1125dca8d
Nov 03 14:27:59.512529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:27:59.512503    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"]
Nov 03 14:28:00.083622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:00.083529    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb" event={"ID":"ea037f26-764f-4aee-818c-698f952bcad9","Type":"ContainerStarted","Data":"bc8193fa0ca97d8adb2ac30b6cebe0d259634d0343933b8c54d511e7ed8b6787"}
Nov 03 14:28:00.083622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:00.083589    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb" event={"ID":"ea037f26-764f-4aee-818c-698f952bcad9","Type":"ContainerStarted","Data":"1b32a92962578ae86763440a512fc4c7f5de4f4c3c225024fb0807d1125dca8d"}
Nov 03 14:28:00.201440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:00.201395    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:01.102262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:01.102209    6481 generic.go:356] "Generic (PLEG): container finished" podID="ea037f26-764f-4aee-818c-698f952bcad9" containerID="bc8193fa0ca97d8adb2ac30b6cebe0d259634d0343933b8c54d511e7ed8b6787" exitCode=0
Nov 03 14:28:01.102701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:01.102264    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb" event={"ID":"ea037f26-764f-4aee-818c-698f952bcad9","Type":"ContainerDied","Data":"bc8193fa0ca97d8adb2ac30b6cebe0d259634d0343933b8c54d511e7ed8b6787"}
Nov 03 14:28:02.227978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.227929    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:02.280629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.280607    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:28:02.335413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.335385    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h\" (UniqueName: \"kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h\") pod \"ea037f26-764f-4aee-818c-698f952bcad9\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") "
Nov 03 14:28:02.335560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.335491    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca\") pod \"ea037f26-764f-4aee-818c-698f952bcad9\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") "
Nov 03 14:28:02.335560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.335548    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca\") pod \"ea037f26-764f-4aee-818c-698f952bcad9\" (UID: \"ea037f26-764f-4aee-818c-698f952bcad9\") "
Nov 03 14:28:02.335982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.335951    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca" pod "ea037f26-764f-4aee-818c-698f952bcad9" (UID: "ea037f26-764f-4aee-818c-698f952bcad9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:02.336165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.336139    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:02.337606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.337583    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h" pod "ea037f26-764f-4aee-818c-698f952bcad9" (UID: "ea037f26-764f-4aee-818c-698f952bcad9"). InnerVolumeSpecName "kube-api-access-5w87h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:02.338027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.338001    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca" pod "ea037f26-764f-4aee-818c-698f952bcad9" (UID: "ea037f26-764f-4aee-818c-698f952bcad9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:28:02.437581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.437556    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea037f26-764f-4aee-818c-698f952bcad9-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:02.437581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.437582    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w87h\" (UniqueName: \"kubernetes.io/projected/ea037f26-764f-4aee-818c-698f952bcad9-kube-api-access-5w87h\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:02.811427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:02.811348    6481 scope.go:122] "RemoveContainer" containerID="7f530e08ee92bcc02837339624e702545c3ea678cd42baf0b905d0b770dc7f61"
Nov 03 14:28:03.118970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:03.118883    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"
Nov 03 14:28:03.118970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:03.118911    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb" event={"ID":"ea037f26-764f-4aee-818c-698f952bcad9","Type":"ContainerDied","Data":"1b32a92962578ae86763440a512fc4c7f5de4f4c3c225024fb0807d1125dca8d"}
Nov 03 14:28:03.118970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:03.118961    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b32a92962578ae86763440a512fc4c7f5de4f4c3c225024fb0807d1125dca8d"
Nov 03 14:28:03.244853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:03.244818    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:03.449475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:03.449435    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:03.863307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:03.855028    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:04.104778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:04.104740    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:04.693253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:04.692332    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:28:05.602255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:05.601825    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:05.829276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:05.828311    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:28:06.036253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.035506    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:06.036253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.035878    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:28:06.061253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.044285    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:28:06.061253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.044611    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:28:06.061253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.044857    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:28:06.085257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.074331    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:28:06.085257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.074394    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:06.109255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.086215    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:28:06.109255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.086366    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:06.595255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.594328    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:06.832253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.831380    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:06.832253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:06.832175    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:28:07.089253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.088611    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Nov 03 14:28:07.098266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.096788    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:28:07.098266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.097162    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:28:07.098266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.097462    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:28:07.098266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.097715    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:28:07.098644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.098340    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:28:07.098644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.098401    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:28:07.098644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.098438    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:28:07.098644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.098472    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:28:07.098644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.098507    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:28:07.192255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.191208    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:28:07.374302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.373077    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": context deadline exceeded"
Nov 03 14:28:07.980253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:07.927368    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:28:08.006739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.005381    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:28:08.172940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.171697    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:08.172940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.172084    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:28:08.172940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.172394    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:28:08.413298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.402430    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:28:08.413298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.402517    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:28:08.425126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.421574    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:28:08.434419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.434393    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:28:08.705982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:08.705896    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:10.894667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:10.894634    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:12.048254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.044469    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"]
Nov 03 14:28:12.057764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.057728    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b138e-5lxpw"]
Nov 03 14:28:12.071081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.071048    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74870f1c-4c34-4eb5-95bd-e154dcf7c39a" path="/var/lib/kubelet/pods/74870f1c-4c34-4eb5-95bd-e154dcf7c39a/volumes"
Nov 03 14:28:12.319930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.319852    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"]
Nov 03 14:28:12.320563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.320543    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ea037f26-764f-4aee-818c-698f952bcad9" containerName="revoke-keys"
Nov 03 14:28:12.320649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.320565    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea037f26-764f-4aee-818c-698f952bcad9" containerName="revoke-keys"
Nov 03 14:28:12.320773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.320759    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea037f26-764f-4aee-818c-698f952bcad9" containerName="revoke-keys"
Nov 03 14:28:12.321398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.321333    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:12.326952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.326934    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.343026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.343002    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"]
Nov 03 14:28:12.350999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.350976    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:28:12.351096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.351080    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:28:12.351169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.351117    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:28:12.351169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.351131    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:28:12.390370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.390336    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.390508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.390446    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.390508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.390494    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.491060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.491034    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.491202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.491141    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.491306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.491203    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.491790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.491762    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.493539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.493521    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.514918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.514896    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b\") pod \"maas-api-revoke-sub-1c676596-j28lc\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.523145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.523111    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:12.640093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.640031    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:12.942172 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:28:12.942141    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88585a1c_f8d9_48d8_81fe_68ac03fd5e2f.slice/crio-b30376f922dc1a00934f76180d31e87046b98968eb1a4aab866c310c0377331c WatchSource:0}: Error finding container b30376f922dc1a00934f76180d31e87046b98968eb1a4aab866c310c0377331c: Status 404 returned error can't find the container with id b30376f922dc1a00934f76180d31e87046b98968eb1a4aab866c310c0377331c
Nov 03 14:28:12.943057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:12.943004    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"]
Nov 03 14:28:13.138252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.138210    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:13.343317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.343276    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:13.432318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.432277    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc" event={"ID":"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f","Type":"ContainerStarted","Data":"e5095e820438e9203275f776482c07ae4b5fcfddd654910af4a4289a2592bc7d"}
Nov 03 14:28:13.432503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.432351    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc" event={"ID":"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f","Type":"ContainerStarted","Data":"b30376f922dc1a00934f76180d31e87046b98968eb1a4aab866c310c0377331c"}
Nov 03 14:28:13.478366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.478278    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc" podStartSLOduration=1.478264322 podStartE2EDuration="1.478264322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:13.476110018 +0000 UTC m=+971.526141410" watchObservedRunningTime="2026-11-03 14:28:13.478264322 +0000 UTC m=+971.528295502"
Nov 03 14:28:13.589198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.589163    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"]
Nov 03 14:28:13.589500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.589470    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy" containerID="cri-o://56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989" gracePeriod=30
Nov 03 14:28:13.719254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:13.709894    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:14.441729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.441694    6481 generic.go:356] "Generic (PLEG): container finished" podID="88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" containerID="e5095e820438e9203275f776482c07ae4b5fcfddd654910af4a4289a2592bc7d" exitCode=0
Nov 03 14:28:14.442126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.441741    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc" event={"ID":"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f","Type":"ContainerDied","Data":"e5095e820438e9203275f776482c07ae4b5fcfddd654910af4a4289a2592bc7d"}
Nov 03 14:28:14.450489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.450464    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"]
Nov 03 14:28:14.454556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.454534    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.473317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.473294    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dmczr\""
Nov 03 14:28:14.475902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.475878    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 14:28:14.476018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.475960    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 14:28:14.476018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.475961    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 14:28:14.478285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.478265    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"]
Nov 03 14:28:14.514969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.514943    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.515133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.514995    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7rsm\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.515133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.515112    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.515292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.515139    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.515292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.515218    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.515292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.515288    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.515457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.515374    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.615943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.615909    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.615943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.615946    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.616135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.615989    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.616135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.616029    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.616135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.616082    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.616135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.616121    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.616354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.616183    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7rsm\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.616775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.616747    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.618318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.618296    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.618588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.618566    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.618635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.618578    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.618673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.618648    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.638337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.638314    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.638815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.638794    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7rsm\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r9t9j\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:14.669984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.669960    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:28:14.767844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:14.767785    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:15.087401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.085785    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"]
Nov 03 14:28:15.094166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.094148    6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:28:15.383197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.383120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:15.450164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.450131    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j" event={"ID":"a25d4e45-29be-4581-b786-68e95371cde6","Type":"ContainerStarted","Data":"a6b640244fe1de327a523bb5d38927a1594dea41eccc867edfe5abd4eed22154"}
Nov 03 14:28:15.590726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.590703    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:15.625286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.625257    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca\") pod \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") "
Nov 03 14:28:15.625491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.625474    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b\") pod \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") "
Nov 03 14:28:15.625542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.625527    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca\") pod \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\" (UID: \"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f\") "
Nov 03 14:28:15.625912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.625888    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca" pod "88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" (UID: "88585a1c-f8d9-48d8-81fe-68ac03fd5e2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:15.626074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.626021    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:15.627258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.627213    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca" pod "88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" (UID: "88585a1c-f8d9-48d8-81fe-68ac03fd5e2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:28:15.627586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.627564    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b" pod "88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" (UID: "88585a1c-f8d9-48d8-81fe-68ac03fd5e2f"). InnerVolumeSpecName "kube-api-access-bcr9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:15.727347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.727316    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcr9b\" (UniqueName: \"kubernetes.io/projected/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-kube-api-access-bcr9b\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:15.727347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:15.727340    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:16.418868 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:16.418835    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:16.461353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:16.461316    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc" event={"ID":"88585a1c-f8d9-48d8-81fe-68ac03fd5e2f","Type":"ContainerDied","Data":"b30376f922dc1a00934f76180d31e87046b98968eb1a4aab866c310c0377331c"}
Nov 03 14:28:16.461710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:16.461379    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"
Nov 03 14:28:16.461710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:16.461379    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b30376f922dc1a00934f76180d31e87046b98968eb1a4aab866c310c0377331c"
Nov 03 14:28:16.638975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:16.638945    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:17.082797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:17.082764    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:17.456266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:17.456201    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:17.958916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:17.958881    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:28:17.959331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:17.958958    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:28:18.484518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.484479    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j" event={"ID":"a25d4e45-29be-4581-b786-68e95371cde6","Type":"ContainerStarted","Data":"da721857c7598744f60fef2bcfb50aede42d1210a808c374a54b019d17800cd5"}
Nov 03 14:28:18.484957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.484934    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:18.561749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.561696    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j" podStartSLOduration=1.698151617 podStartE2EDuration="4.561680302s" totalImagesPullingTime="2.863528685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:18.54541786 +0000 UTC m=+976.595449056" watchObservedRunningTime="2026-11-03 14:28:18.561680302 +0000 UTC m=+976.611711482"
Nov 03 14:28:18.703109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.703074    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:28:18.909386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.909359    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:28:18.969133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969089    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969260    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969357    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969406    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969512    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969531    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:28:18.969614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969541    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969629    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969687    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.969952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969731    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token\") pod \"19dd7a32-708d-43ca-97a3-29f8d9d77578\" (UID: \"19dd7a32-708d-43ca-97a3-29f8d9d77578\") "
Nov 03 14:28:18.970104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.969966    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:18.970327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.970305    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:18.970411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.970333    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19dd7a32-708d-43ca-97a3-29f8d9d77578-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:18.970611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.970582    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:28:18.970686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.970647    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:28:18.970751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.970693    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:28:18.972697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.972669    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:28:18.973262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.973160    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:18.973262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.973217    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "kube-api-access-4hjbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:18.976645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:18.976619    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy" pod "19dd7a32-708d-43ca-97a3-29f8d9d77578" (UID: "19dd7a32-708d-43ca-97a3-29f8d9d77578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:28:19.072287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072191    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.072287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072244    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.072287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072260    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hjbx\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-kube-api-access-4hjbx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.072287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072277    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.072287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072291    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.072641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072305    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.072641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.072318    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19dd7a32-708d-43ca-97a3-29f8d9d77578-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:19.725690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.725652    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:19.764954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.764477    6481 generic.go:356] "Generic (PLEG): container finished" podID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerID="e8da05c80a84a07f5bd00f250fe09b6ae1427d84467f677637f8106df66453c2" exitCode=0
Nov 03 14:28:19.764954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.764829    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" event={"ID":"ef5c42db-c9e0-49c0-b008-33d595ea3c78","Type":"ContainerDied","Data":"e8da05c80a84a07f5bd00f250fe09b6ae1427d84467f677637f8106df66453c2"}
Nov 03 14:28:19.838614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.821932    6481 generic.go:356] "Generic (PLEG): container finished" podID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerID="56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989" exitCode=0
Nov 03 14:28:19.838614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.833154    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"
Nov 03 14:28:19.857334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.857288    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" event={"ID":"19dd7a32-708d-43ca-97a3-29f8d9d77578","Type":"ContainerDied","Data":"56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989"}
Nov 03 14:28:19.866261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.860333    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp" event={"ID":"19dd7a32-708d-43ca-97a3-29f8d9d77578","Type":"ContainerDied","Data":"f30d9857aee6f0d0ae3818cb73be27852e3285920a00921b27ac8a93486456de"}
Nov 03 14:28:19.866261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.860405    6481 scope.go:122] "RemoveContainer" containerID="56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989"
Nov 03 14:28:19.907770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:19.888550    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:28:20.419249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.415472    6481 scope.go:122] "RemoveContainer" containerID="56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989"
Nov 03 14:28:20.419249 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:28:20.416970    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989\": container with ID starting with 56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989 not found: ID does not exist" containerID="56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989"
Nov 03 14:28:20.419249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.417010    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989"} err="failed to get container status \"56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989\": rpc error: code = NotFound desc = could not find container \"56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989\": container with ID starting with 56848c6fdcaa895c3a9a12a98d9b02782e3d2459d5e20ee66fdac75fc0a53989 not found: ID does not exist"
Nov 03 14:28:20.458254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.456004    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:28:20.500282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.499492    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"]
Nov 03 14:28:20.513279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.507739    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc\" (UniqueName: \"kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc\") pod \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") "
Nov 03 14:28:20.513279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.507920    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume\") pod \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\" (UID: \"ef5c42db-c9e0-49c0-b008-33d595ea3c78\") "
Nov 03 14:28:20.513279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.508696    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume" pod "ef5c42db-c9e0-49c0-b008-33d595ea3c78" (UID: "ef5c42db-c9e0-49c0-b008-33d595ea3c78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:20.550253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.543718    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e759e5-openshift-default-765cc7f4f-cncgp"]
Nov 03 14:28:20.579252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.578426    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc" pod "ef5c42db-c9e0-49c0-b008-33d595ea3c78" (UID: "ef5c42db-c9e0-49c0-b008-33d595ea3c78"). InnerVolumeSpecName "kube-api-access-cp8lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:20.661565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.661504    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5c42db-c9e0-49c0-b008-33d595ea3c78-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:20.661565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.661538    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp8lc\" (UniqueName: \"kubernetes.io/projected/ef5c42db-c9e0-49c0-b008-33d595ea3c78-kube-api-access-cp8lc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:20.862313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.860787    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"
Nov 03 14:28:20.874264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.872396    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f" event={"ID":"ef5c42db-c9e0-49c0-b008-33d595ea3c78","Type":"ContainerDied","Data":"50ede6932b37308e0c756d14409f06d09753e6b05a02f6fbf1a322542f2c5260"}
Nov 03 14:28:20.874264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:20.872468    6481 scope.go:122] "RemoveContainer" containerID="e8da05c80a84a07f5bd00f250fe09b6ae1427d84467f677637f8106df66453c2"
Nov 03 14:28:21.005651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:21.005618    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"]
Nov 03 14:28:21.037261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:21.030769    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-e759e5-6c98644d9f-sls9f"]
Nov 03 14:28:21.887255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:21.886318    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:22.000271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:21.999347    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:28:22.010263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.008305    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:22.144307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.143556    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" path="/var/lib/kubelet/pods/19dd7a32-708d-43ca-97a3-29f8d9d77578/volumes"
Nov 03 14:28:22.144307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.144174    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" path="/var/lib/kubelet/pods/ef5c42db-c9e0-49c0-b008-33d595ea3c78/volumes"
Nov 03 14:28:22.177254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.176325    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:22.198465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.198424    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Nov 03 14:28:22.412314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.408338    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:22.448251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:22.447877    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:23.000797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.000753    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:23.005809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.005784    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:28:23.089251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.084785    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:28:23.178362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.176651    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:23.178362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.177367    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:28:23.413509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.413460    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:23.427276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.425101    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:28:23.460150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.460111    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:28:23.461010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.460985    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:28:23.646968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.633668    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:28:23.689318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.671463    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:28:23.783849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.783818    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"]
Nov 03 14:28:23.786307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.786284    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerName="payload-pre-processing"
Nov 03 14:28:23.786575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.786558    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerName="payload-pre-processing"
Nov 03 14:28:23.786740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.786727    6481 cpu_manager.go:414] "containerMap: removing container" podUID="88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" containerName="revoke-keys"
Nov 03 14:28:23.786954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.786940    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" containerName="revoke-keys"
Nov 03 14:28:23.787171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.787158    6481 cpu_manager.go:414] "containerMap: removing container" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy"
Nov 03 14:28:23.787425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.787408    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy"
Nov 03 14:28:23.787906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.787888    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dd7a32-708d-43ca-97a3-29f8d9d77578" containerName="istio-proxy"
Nov 03 14:28:23.788038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.788026    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5c42db-c9e0-49c0-b008-33d595ea3c78" containerName="payload-pre-processing"
Nov 03 14:28:23.788139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.788127    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" containerName="revoke-keys"
Nov 03 14:28:23.808898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.808871    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:28:23.810260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.810164    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:23.815371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.815340    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"]
Nov 03 14:28:23.852365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.852096    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:28:23.853572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.853546    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:28:23.853717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.853643    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:28:23.853787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.853737    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:28:23.914390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.914362    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:23.966871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.966771    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:23.966871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.966827    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-472jn\" (UniqueName: \"kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:23.967108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:23.966928    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.068207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.068174    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.068724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.068239    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-472jn\" (UniqueName: \"kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.068724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.068344    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.069258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.069217    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.070822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.070795    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.091305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.091280    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-472jn\" (UniqueName: \"kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn\") pod \"maas-api-revoke-sub-77405b85-5n8gc\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.164883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.164853    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:24.471687 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:28:24.471606    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3f021a_a57c_40e2_a672_72a60b5e4b82.slice/crio-22ad4165f84415e8250bace8ba1138d59a70196487fe2f88f3bab4d7142d28ee WatchSource:0}: Error finding container 22ad4165f84415e8250bace8ba1138d59a70196487fe2f88f3bab4d7142d28ee: Status 404 returned error can't find the container with id 22ad4165f84415e8250bace8ba1138d59a70196487fe2f88f3bab4d7142d28ee
Nov 03 14:28:24.471687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.471610    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"]
Nov 03 14:28:24.984358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.984291    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc" event={"ID":"ea3f021a-a57c-40e2-a672-72a60b5e4b82","Type":"ContainerStarted","Data":"a538a24ced8e1ae6ce776fec6c3a678095e4f94280116258795fb0a3132436ac"}
Nov 03 14:28:24.984358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:24.984340    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc" event={"ID":"ea3f021a-a57c-40e2-a672-72a60b5e4b82","Type":"ContainerStarted","Data":"22ad4165f84415e8250bace8ba1138d59a70196487fe2f88f3bab4d7142d28ee"}
Nov 03 14:28:25.040757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:25.040707    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc" podStartSLOduration=2.04069344 podStartE2EDuration="2.04069344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:25.026910911 +0000 UTC m=+983.076942094" watchObservedRunningTime="2026-11-03 14:28:25.04069344 +0000 UTC m=+983.090724623"
Nov 03 14:28:25.754662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:25.754619    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:25.962996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:25.962961    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:25.993485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:25.993456    6481 generic.go:356] "Generic (PLEG): container finished" podID="ea3f021a-a57c-40e2-a672-72a60b5e4b82" containerID="a538a24ced8e1ae6ce776fec6c3a678095e4f94280116258795fb0a3132436ac" exitCode=0
Nov 03 14:28:25.993657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:25.993541    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc" event={"ID":"ea3f021a-a57c-40e2-a672-72a60b5e4b82","Type":"ContainerDied","Data":"a538a24ced8e1ae6ce776fec6c3a678095e4f94280116258795fb0a3132436ac"}
Nov 03 14:28:26.185351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:26.185319    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:26.404987 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:26.404937    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:26.757249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:26.757201    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:26.963390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:26.963356    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:27.154065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.154035    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:27.311027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.310943    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn\" (UniqueName: \"kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn\") pod \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") "
Nov 03 14:28:27.311194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.311068    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca\") pod \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") "
Nov 03 14:28:27.311194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.311116    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca\") pod \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\" (UID: \"ea3f021a-a57c-40e2-a672-72a60b5e4b82\") "
Nov 03 14:28:27.311494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.311466    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca" pod "ea3f021a-a57c-40e2-a672-72a60b5e4b82" (UID: "ea3f021a-a57c-40e2-a672-72a60b5e4b82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:27.311608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.311594    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:27.313028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.313000    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn" pod "ea3f021a-a57c-40e2-a672-72a60b5e4b82" (UID: "ea3f021a-a57c-40e2-a672-72a60b5e4b82"). InnerVolumeSpecName "kube-api-access-472jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:27.313120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.313074    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca" pod "ea3f021a-a57c-40e2-a672-72a60b5e4b82" (UID: "ea3f021a-a57c-40e2-a672-72a60b5e4b82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:28:27.413085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.413055    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea3f021a-a57c-40e2-a672-72a60b5e4b82-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:27.413085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:27.413084    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-472jn\" (UniqueName: \"kubernetes.io/projected/ea3f021a-a57c-40e2-a672-72a60b5e4b82-kube-api-access-472jn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:27.985803 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:28:27.985766    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:28:28.011202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:28.011176    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"
Nov 03 14:28:28.011391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:28.011271    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc" event={"ID":"ea3f021a-a57c-40e2-a672-72a60b5e4b82","Type":"ContainerDied","Data":"22ad4165f84415e8250bace8ba1138d59a70196487fe2f88f3bab4d7142d28ee"}
Nov 03 14:28:28.011391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:28.011359    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ad4165f84415e8250bace8ba1138d59a70196487fe2f88f3bab4d7142d28ee"
Nov 03 14:28:28.306114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:28.306030    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:29.664346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:29.664313    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:29.869766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:29.869735    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:30.738806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:30.738775    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:31.526580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:31.526546    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:31.779131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:31.767698    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:33.010494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:33.010462    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:33.226324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:33.226275    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:35.273693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:35.273657    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:36.565827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:36.565792    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:36.773711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:36.773677    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:37.603870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:37.603836    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:39.379129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.379086    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:39.522383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.522353    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"]
Nov 03 14:28:39.522922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.522904    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ea3f021a-a57c-40e2-a672-72a60b5e4b82" containerName="revoke-keys"
Nov 03 14:28:39.522922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.522920    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3f021a-a57c-40e2-a672-72a60b5e4b82" containerName="revoke-keys"
Nov 03 14:28:39.523095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.523060    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3f021a-a57c-40e2-a672-72a60b5e4b82" containerName="revoke-keys"
Nov 03 14:28:39.528107 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.528089    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.546889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.546865    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"]
Nov 03 14:28:39.547915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.547896    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-c7f4d47a6133\""
Nov 03 14:28:39.554928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.554903    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-zh4h2\""
Nov 03 14:28:39.555552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.555528    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\""
Nov 03 14:28:39.555552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.555534    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\""
Nov 03 14:28:39.560790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.560772    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\""
Nov 03 14:28:39.560905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.560825    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\""
Nov 03 14:28:39.631020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.630927    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.631172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.631074    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzlts\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.631217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.631204    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.631366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.631331    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.637425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.637399    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:39.732634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.732591    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.732818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.732679    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzlts\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.733015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.732991    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.733121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.733101    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.735401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.734550    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.736154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.736128    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.761567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.757956    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.767270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.767210    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzlts\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:39.842858 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:39.842821    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:40.178469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:40.178440    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"]
Nov 03 14:28:41.050054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.050019    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"]
Nov 03 14:28:41.062510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.062483    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b46debb-gpp8f"]
Nov 03 14:28:41.184650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.184610    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq" event={"ID":"64181889-5c5a-4059-bc71-3676b91efcd7","Type":"ContainerStarted","Data":"bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb"}
Nov 03 14:28:41.184838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.184681    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:41.184838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.184707    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq" event={"ID":"64181889-5c5a-4059-bc71-3676b91efcd7","Type":"ContainerStarted","Data":"af7dbfc5e20b90079db1b2ae397ba610498b69d264c2115846024e717805e6ad"}
Nov 03 14:28:41.233545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.233492    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq" podStartSLOduration=2.233478267 podStartE2EDuration="2.233478267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:28:41.231854961 +0000 UTC m=+999.281886144" watchObservedRunningTime="2026-11-03 14:28:41.233478267 +0000 UTC m=+999.283509449"
Nov 03 14:28:41.700911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:41.700872    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:42.067897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:42.067825    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0360855e-b5e0-4bbe-b754-0dc3a4d34a0f" path="/var/lib/kubelet/pods/0360855e-b5e0-4bbe-b754-0dc3a4d34a0f/volumes"
Nov 03 14:28:43.713193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:43.713077    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:43.932851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:43.932823    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:44.147598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:44.147508    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:44.299013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:44.298975    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"]
Nov 03 14:28:44.472183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:44.472146    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:45.915149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:45.915116    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:28:46.105296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:46.105264    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:46.506738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:46.506703    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:46.772785 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:46.772703    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:48.624254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:48.620177    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:49.299839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.299784    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq" podUID="64181889-5c5a-4059-bc71-3676b91efcd7" containerName="payload-processing" containerID="cri-o://bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb" gracePeriod=25
Nov 03 14:28:49.571243 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.571209    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:49.655126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.655099    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials\") pod \"64181889-5c5a-4059-bc71-3676b91efcd7\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") "
Nov 03 14:28:49.655545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.655150    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume\") pod \"64181889-5c5a-4059-bc71-3676b91efcd7\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") "
Nov 03 14:28:49.655545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.655240    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts\") pod \"64181889-5c5a-4059-bc71-3676b91efcd7\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") "
Nov 03 14:28:49.655545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.655307    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume\") pod \"64181889-5c5a-4059-bc71-3676b91efcd7\" (UID: \"64181889-5c5a-4059-bc71-3676b91efcd7\") "
Nov 03 14:28:49.655884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.655857    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume" pod "64181889-5c5a-4059-bc71-3676b91efcd7" (UID: "64181889-5c5a-4059-bc71-3676b91efcd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:49.655981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.655899    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume" pod "64181889-5c5a-4059-bc71-3676b91efcd7" (UID: "64181889-5c5a-4059-bc71-3676b91efcd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:28:49.656067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.656049    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:49.656117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.656074    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64181889-5c5a-4059-bc71-3676b91efcd7-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:49.657766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.657744    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts" pod "64181889-5c5a-4059-bc71-3676b91efcd7" (UID: "64181889-5c5a-4059-bc71-3676b91efcd7"). InnerVolumeSpecName "kube-api-access-wzlts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:49.657877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.657849    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials" pod "64181889-5c5a-4059-bc71-3676b91efcd7" (UID: "64181889-5c5a-4059-bc71-3676b91efcd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:28:49.756778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.756745    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:49.756778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:49.756774    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzlts\" (UniqueName: \"kubernetes.io/projected/64181889-5c5a-4059-bc71-3676b91efcd7-kube-api-access-wzlts\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:28:50.302244 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.302201    6481 generic.go:356] "Generic (PLEG): container finished" podID="64181889-5c5a-4059-bc71-3676b91efcd7" containerID="bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb" exitCode=0
Nov 03 14:28:50.302425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.302261    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"
Nov 03 14:28:50.302425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.302290    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq" event={"ID":"64181889-5c5a-4059-bc71-3676b91efcd7","Type":"ContainerDied","Data":"bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb"}
Nov 03 14:28:50.302425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.302341    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq" event={"ID":"64181889-5c5a-4059-bc71-3676b91efcd7","Type":"ContainerDied","Data":"af7dbfc5e20b90079db1b2ae397ba610498b69d264c2115846024e717805e6ad"}
Nov 03 14:28:50.302425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.302362    6481 scope.go:122] "RemoveContainer" containerID="bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb"
Nov 03 14:28:50.312671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.312648    6481 scope.go:122] "RemoveContainer" containerID="bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb"
Nov 03 14:28:50.312937 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:28:50.312920    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb\": container with ID starting with bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb not found: ID does not exist" containerID="bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb"
Nov 03 14:28:50.312980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.312945    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb"} err="failed to get container status \"bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb\": rpc error: code = NotFound desc = could not find container \"bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb\": container with ID starting with bd2892944f000e03bc0c70fd614d426a3eaaacd44e701c55b58dfb59cb8eefcb not found: ID does not exist"
Nov 03 14:28:50.365567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.365513    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"]
Nov 03 14:28:50.378377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.378354    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-6984k7shq"]
Nov 03 14:28:50.712031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:50.711988    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:52.068783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:52.068751    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64181889-5c5a-4059-bc71-3676b91efcd7" path="/var/lib/kubelet/pods/64181889-5c5a-4059-bc71-3676b91efcd7/volumes"
Nov 03 14:28:52.724413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:52.724386    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:54.684549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:54.684501    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:56.649776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:56.649743    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:57.082793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:57.082757    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"]
Nov 03 14:28:57.088049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:57.088025    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7556663-kn22k"]
Nov 03 14:28:57.746242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:57.746208    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:57.986640 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:28:57.986603    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:28:58.069875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:58.069792    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e23ad60-fc64-49dd-98ca-2e65a72d9882" path="/var/lib/kubelet/pods/8e23ad60-fc64-49dd-98ca-2e65a72d9882/volumes"
Nov 03 14:28:58.507909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:58.507878    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:58.727725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:58.727677    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:58.936097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:58.936057    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:59.152183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:59.152147    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:59.746639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:59.746604    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:28:59.974259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:28:59.974141    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:01.358877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:01.358779    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:01.600158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:01.600114    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:02.023257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:02.021325    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:02.104743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:02.094938    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:02.644759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:02.644701    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:02.717372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:02.717316    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:02.812430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:02.812381    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:03.097845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.097799    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:29:03.125270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.111983    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:29:03.202609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.202502    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:03.406255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.398933    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:03.663326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.654473    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:03.663326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.656794    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:29:03.663326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.658181    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:29:03.724255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.720450    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:03.818256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.754266    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:03.850462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.785949    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:29:04.171315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.091363    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:29:04.171315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:03.850203    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:29:04.185253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.184575    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:29:04.190252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.186871    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:29:04.234217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.214655    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:04.307569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.297144    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:04.447137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.440405    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"]
Nov 03 14:29:04.452929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.452897    6481 cpu_manager.go:414] "containerMap: removing container" podUID="64181889-5c5a-4059-bc71-3676b91efcd7" containerName="payload-processing"
Nov 03 14:29:04.453143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.453124    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64181889-5c5a-4059-bc71-3676b91efcd7" containerName="payload-processing"
Nov 03 14:29:04.453533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.453511    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="64181889-5c5a-4059-bc71-3676b91efcd7" containerName="payload-processing"
Nov 03 14:29:04.485537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.485506    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:29:04.486651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.486628    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.508276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.508248    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-b930455f941e\""
Nov 03 14:29:04.583048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.582968    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"]
Nov 03 14:29:04.611088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.603045    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.611088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.603168    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.611088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.603214    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.611088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.603335    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l77rr\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.671616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.665408    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:04.707615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.707527    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.710519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.710303    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.731801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.731775    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.738737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.738710    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l77rr\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.768040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.728839    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.768465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.738019    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.826168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.766664    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:04.826983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.798470    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.844998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.844961    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l77rr\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:04.941273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.941224    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:29:04.978962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:04.978870    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:05.161251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.154325    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:29:05.211351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.210996    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:29:05.219622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.212333    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:29:05.219622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.212392    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:05.221970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.221946    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:29:05.222487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.222429    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:29:05.239295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.230515    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:29:05.515043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.514760    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:05.844131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:05.844052    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"]
Nov 03 14:29:05.849480 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:29:05.849455    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfb3ee4_75e8_4a9f_bfd0_721429024e02.slice/crio-2e8b9620a388952f6b7dc8a9aa023657769c4ea0d69f03ccc14f884f07fdff37 WatchSource:0}: Error finding container 2e8b9620a388952f6b7dc8a9aa023657769c4ea0d69f03ccc14f884f07fdff37: Status 404 returned error can't find the container with id 2e8b9620a388952f6b7dc8a9aa023657769c4ea0d69f03ccc14f884f07fdff37
Nov 03 14:29:06.609366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:06.609335    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:06.647826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:06.647796    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b" event={"ID":"7dfb3ee4-75e8-4a9f-bfd0-721429024e02","Type":"ContainerStarted","Data":"3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781"}
Nov 03 14:29:06.647954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:06.647842    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b" event={"ID":"7dfb3ee4-75e8-4a9f-bfd0-721429024e02","Type":"ContainerStarted","Data":"2e8b9620a388952f6b7dc8a9aa023657769c4ea0d69f03ccc14f884f07fdff37"}
Nov 03 14:29:06.648022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:06.647978    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:06.694388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:06.694340    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b" podStartSLOduration=2.694327627 podStartE2EDuration="2.694327627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:06.69231365 +0000 UTC m=+1024.742344824" watchObservedRunningTime="2026-11-03 14:29:06.694327627 +0000 UTC m=+1024.744358809"
Nov 03 14:29:07.132895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:07.132864    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:07.352812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:07.352755    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:07.943038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:07.943006    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:10.045795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:10.045764    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"]
Nov 03 14:29:10.058057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:10.058030    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-676e120e-twb8b"]
Nov 03 14:29:10.068476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:10.068446    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83022cb0-9424-4629-bb1f-464c772af24a" path="/var/lib/kubelet/pods/83022cb0-9424-4629-bb1f-464c772af24a/volumes"
Nov 03 14:29:10.236627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:10.236591    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:10.485576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:10.485543    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:10.715407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:10.715378    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:12.385650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.385614    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:12.420721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.420693    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"]
Nov 03 14:29:12.426039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.426024    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.446547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.446525    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"]
Nov 03 14:29:12.454881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.454858    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4a67e25f\""
Nov 03 14:29:12.455000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.454880    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4a67e25f-dockercfg-hsrd7\""
Nov 03 14:29:12.588146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.588110    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.588387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.588317    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmq2c\" (UniqueName: \"kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.596313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.596283    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:12.689382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.689351    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.689562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.689445    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmq2c\" (UniqueName: \"kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.690289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.690266    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.712060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.712019    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmq2c\" (UniqueName: \"kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c\") pod \"payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.739729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.739697    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:12.817939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:12.817782    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:13.048311 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:29:13.048271    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169973d6_32ab_4a5a_ba21_7310d53d8493.slice/crio-0ec393920fc173dc862818c6b03d253ad8088bd405b179790f3875ff9afc31ae WatchSource:0}: Error finding container 0ec393920fc173dc862818c6b03d253ad8088bd405b179790f3875ff9afc31ae: Status 404 returned error can't find the container with id 0ec393920fc173dc862818c6b03d253ad8088bd405b179790f3875ff9afc31ae
Nov 03 14:29:13.048417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:13.048405    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"]
Nov 03 14:29:13.701283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:13.701223    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" event={"ID":"169973d6-32ab-4a5a-ba21-7310d53d8493","Type":"ContainerStarted","Data":"5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be"}
Nov 03 14:29:13.701748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:13.701318    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:13.701748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:13.701343    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" event={"ID":"169973d6-32ab-4a5a-ba21-7310d53d8493","Type":"ContainerStarted","Data":"0ec393920fc173dc862818c6b03d253ad8088bd405b179790f3875ff9afc31ae"}
Nov 03 14:29:13.703700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:13.703677    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:29:13.751395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:13.751344    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" podStartSLOduration=1.7513301129999999 podStartE2EDuration="1.751330113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:13.748353278 +0000 UTC m=+1031.798384462" watchObservedRunningTime="2026-11-03 14:29:13.751330113 +0000 UTC m=+1031.801361295"
Nov 03 14:29:15.038574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:15.038534    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:15.842470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:15.834444    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded"
Nov 03 14:29:16.021253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:16.013330    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:16.176427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:16.176386    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:29:16.200372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:16.200333    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:16.234028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:16.233991    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:29:17.424057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:17.424017    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:17.665258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:17.663987    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:17.751826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:17.751747    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"]
Nov 03 14:29:19.595856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:19.595823    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:21.673582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.673551    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"]
Nov 03 14:29:21.678165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.678146    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.700845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.700817    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"]
Nov 03 14:29:21.706592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.706567    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:29:21.706818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.706801    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:29:21.706983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.706960    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:29:21.707100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.707083    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:29:21.711110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.711077    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:21.805763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.805724    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.805930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.805773    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd8r2\" (UniqueName: \"kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.805982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.805936    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.907084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.907050    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.907084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.907086    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd8r2\" (UniqueName: \"kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.907318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.907174    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.907914 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.907888    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.909511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.909490    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.929347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.929292    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd8r2\" (UniqueName: \"kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2\") pod \"maas-api-revoke-sub-42aeeda5-76767\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:21.998366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:21.998329    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:22.134366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.134200    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:22.313350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.313318    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"]
Nov 03 14:29:22.313494 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:29:22.313383    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83a9494_63b1_430f_b828_a59664bb53b2.slice/crio-1a54b5aac4787dde7c566fa14efe951e6b5b828cd73110296269660b6575f9b4 WatchSource:0}: Error finding container 1a54b5aac4787dde7c566fa14efe951e6b5b828cd73110296269660b6575f9b4: Status 404 returned error can't find the container with id 1a54b5aac4787dde7c566fa14efe951e6b5b828cd73110296269660b6575f9b4
Nov 03 14:29:22.350253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.350220    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:22.752921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.752878    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9" podUID="f5b1c21a-bb43-48c8-9add-376c566a87c6" containerName="payload-processing" containerID="cri-o://50e24e964eb19e796e23ef366fcf6031f3165be2ae8abdee3b423931f49e8757" gracePeriod=25
Nov 03 14:29:22.889067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.889028    6481 generic.go:356] "Generic (PLEG): container finished" podID="f5b1c21a-bb43-48c8-9add-376c566a87c6" containerID="50e24e964eb19e796e23ef366fcf6031f3165be2ae8abdee3b423931f49e8757" exitCode=0
Nov 03 14:29:22.889212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.889095    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9" event={"ID":"f5b1c21a-bb43-48c8-9add-376c566a87c6","Type":"ContainerDied","Data":"50e24e964eb19e796e23ef366fcf6031f3165be2ae8abdee3b423931f49e8757"}
Nov 03 14:29:22.890915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.890891    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767" event={"ID":"b83a9494-63b1-430f-b828-a59664bb53b2","Type":"ContainerStarted","Data":"5677c57d864c782dc9d7da178ea555639b814bda440e8990cd878affe7880cd2"}
Nov 03 14:29:22.891026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.890945    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767" event={"ID":"b83a9494-63b1-430f-b828-a59664bb53b2","Type":"ContainerStarted","Data":"1a54b5aac4787dde7c566fa14efe951e6b5b828cd73110296269660b6575f9b4"}
Nov 03 14:29:22.935994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:22.935939    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767" podStartSLOduration=1.935923062 podStartE2EDuration="1.935923062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:22.9323744 +0000 UTC m=+1040.982405584" watchObservedRunningTime="2026-11-03 14:29:22.935923062 +0000 UTC m=+1040.985954246"
Nov 03 14:29:23.018329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.018305    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:29:23.125112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125067    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume\") pod \"f5b1c21a-bb43-48c8-9add-376c566a87c6\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") "
Nov 03 14:29:23.125305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125136    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume\") pod \"f5b1c21a-bb43-48c8-9add-376c566a87c6\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") "
Nov 03 14:29:23.125305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125172    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials\") pod \"f5b1c21a-bb43-48c8-9add-376c566a87c6\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") "
Nov 03 14:29:23.125395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125326    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2\") pod \"f5b1c21a-bb43-48c8-9add-376c566a87c6\" (UID: \"f5b1c21a-bb43-48c8-9add-376c566a87c6\") "
Nov 03 14:29:23.125595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125572    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume" pod "f5b1c21a-bb43-48c8-9add-376c566a87c6" (UID: "f5b1c21a-bb43-48c8-9add-376c566a87c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:23.125673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125584    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume" pod "f5b1c21a-bb43-48c8-9add-376c566a87c6" (UID: "f5b1c21a-bb43-48c8-9add-376c566a87c6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:23.125850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125833    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:23.125889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.125855    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f5b1c21a-bb43-48c8-9add-376c566a87c6-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:23.127552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.127531    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials" pod "f5b1c21a-bb43-48c8-9add-376c566a87c6" (UID: "f5b1c21a-bb43-48c8-9add-376c566a87c6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:23.127610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.127564    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2" pod "f5b1c21a-bb43-48c8-9add-376c566a87c6" (UID: "f5b1c21a-bb43-48c8-9add-376c566a87c6"). InnerVolumeSpecName "kube-api-access-w7cr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:23.226566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.226499    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:23.226566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.226524    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cr2\" (UniqueName: \"kubernetes.io/projected/f5b1c21a-bb43-48c8-9add-376c566a87c6-kube-api-access-w7cr2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:23.796754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.796723    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:23.903918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.903885    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"
Nov 03 14:29:23.904090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.903888    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9" event={"ID":"f5b1c21a-bb43-48c8-9add-376c566a87c6","Type":"ContainerDied","Data":"fc207a184243307527fa4c198ad3fe8589e6f7a64bfeb28a0bef297270308cdd"}
Nov 03 14:29:23.904090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.904057    6481 scope.go:122] "RemoveContainer" containerID="50e24e964eb19e796e23ef366fcf6031f3165be2ae8abdee3b423931f49e8757"
Nov 03 14:29:23.905943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.905917    6481 generic.go:356] "Generic (PLEG): container finished" podID="b83a9494-63b1-430f-b828-a59664bb53b2" containerID="5677c57d864c782dc9d7da178ea555639b814bda440e8990cd878affe7880cd2" exitCode=0
Nov 03 14:29:23.906061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.906035    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767" event={"ID":"b83a9494-63b1-430f-b828-a59664bb53b2","Type":"ContainerDied","Data":"5677c57d864c782dc9d7da178ea555639b814bda440e8990cd878affe7880cd2"}
Nov 03 14:29:23.962780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.962754    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"]
Nov 03 14:29:23.973597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:23.973576    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5q8sb9"]
Nov 03 14:29:24.053820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:24.053717    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"]
Nov 03 14:29:24.058411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:24.058385    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2636454a-qs9zg"]
Nov 03 14:29:24.068437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:24.068413    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43129c35-2401-4252-b4ce-fe074b85ede5" path="/var/lib/kubelet/pods/43129c35-2401-4252-b4ce-fe074b85ede5/volumes"
Nov 03 14:29:24.068834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:24.068822    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b1c21a-bb43-48c8-9add-376c566a87c6" path="/var/lib/kubelet/pods/f5b1c21a-bb43-48c8-9add-376c566a87c6/volumes"
Nov 03 14:29:24.720955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:24.720929    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:29:25.066305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.066282    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:25.249906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.249877    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca\") pod \"b83a9494-63b1-430f-b828-a59664bb53b2\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") "
Nov 03 14:29:25.250057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.250029    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2\" (UniqueName: \"kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2\") pod \"b83a9494-63b1-430f-b828-a59664bb53b2\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") "
Nov 03 14:29:25.250057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.250054    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca\") pod \"b83a9494-63b1-430f-b828-a59664bb53b2\" (UID: \"b83a9494-63b1-430f-b828-a59664bb53b2\") "
Nov 03 14:29:25.250220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.250199    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca" pod "b83a9494-63b1-430f-b828-a59664bb53b2" (UID: "b83a9494-63b1-430f-b828-a59664bb53b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:25.250645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.250613    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:25.252304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.252279    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca" pod "b83a9494-63b1-430f-b828-a59664bb53b2" (UID: "b83a9494-63b1-430f-b828-a59664bb53b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:29:25.252304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.252275    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2" pod "b83a9494-63b1-430f-b828-a59664bb53b2" (UID: "b83a9494-63b1-430f-b828-a59664bb53b2"). InnerVolumeSpecName "kube-api-access-jd8r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:25.351848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.351761    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd8r2\" (UniqueName: \"kubernetes.io/projected/b83a9494-63b1-430f-b828-a59664bb53b2-kube-api-access-jd8r2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:25.351848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.351794    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b83a9494-63b1-430f-b828-a59664bb53b2-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:25.835926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.835897    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:25.925110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.925082    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"
Nov 03 14:29:25.925296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.925112    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767" event={"ID":"b83a9494-63b1-430f-b828-a59664bb53b2","Type":"ContainerDied","Data":"1a54b5aac4787dde7c566fa14efe951e6b5b828cd73110296269660b6575f9b4"}
Nov 03 14:29:25.925296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:25.925172    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a54b5aac4787dde7c566fa14efe951e6b5b828cd73110296269660b6575f9b4"
Nov 03 14:29:27.845691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:27.845651    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:27.988995 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:29:27.988955    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:29:28.065378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:28.065265    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:28.298251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:28.293832    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:30.475381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.470368    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"]
Nov 03 14:29:30.489411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.489382    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f5b1c21a-bb43-48c8-9add-376c566a87c6" containerName="payload-processing"
Nov 03 14:29:30.489619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.489602    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b1c21a-bb43-48c8-9add-376c566a87c6" containerName="payload-processing"
Nov 03 14:29:30.489736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.489723    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b83a9494-63b1-430f-b828-a59664bb53b2" containerName="revoke-keys"
Nov 03 14:29:30.489836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.489823    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b83a9494-63b1-430f-b828-a59664bb53b2" containerName="revoke-keys"
Nov 03 14:29:30.490126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.490107    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b1c21a-bb43-48c8-9add-376c566a87c6" containerName="payload-processing"
Nov 03 14:29:30.490278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.490265    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b83a9494-63b1-430f-b828-a59664bb53b2" containerName="revoke-keys"
Nov 03 14:29:30.515252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.512938    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"]
Nov 03 14:29:30.515252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.513068    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.659580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.646782    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-8809474a8c51\""
Nov 03 14:29:30.660790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.660758    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7tt\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.660997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.660977    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.661158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.661140    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.661472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.661450    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.769510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.768464    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.769857 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.769460    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.770070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.770045    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7tt\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.770758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.770733    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.775377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.775353    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.779377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.776301    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.850007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.849971    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:30.884052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.883975    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:30.886839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:30.886785    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7tt\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:31.015055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.014623    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:29:31.177252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.176886    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:31.511006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.510973    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"]
Nov 03 14:29:31.517269 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:29:31.517218    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770bc1b5_ca92_42a8_874f_da444a26c0f4.slice/crio-aa7964494ed01b455c41041b53b4b10870a6ee22155c57d12910a8afe4195ce6 WatchSource:0}: Error finding container aa7964494ed01b455c41041b53b4b10870a6ee22155c57d12910a8afe4195ce6: Status 404 returned error can't find the container with id aa7964494ed01b455c41041b53b4b10870a6ee22155c57d12910a8afe4195ce6
Nov 03 14:29:31.688301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.687904    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"]
Nov 03 14:29:31.697166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.697137    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"]
Nov 03 14:29:31.697377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.697285    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.715095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715066    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:29:31.715279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715223    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:29:31.715361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715325    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:29:31.715550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715499    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.715708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715585    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:29:31.715708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715593    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.715832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.715730    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstss\" (UniqueName: \"kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.817837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.817735    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.818110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.818085    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.818300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.818280    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstss\" (UniqueName: \"kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.819582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.819556    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.825247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.824351    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:31.839909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:31.839880    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstss\" (UniqueName: \"kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss\") pod \"maas-api-revoke-sub-59591aa0-9zrzl\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:32.019524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.019486    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:32.054458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.054253    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt" event={"ID":"770bc1b5-ca92-42a8-874f-da444a26c0f4","Type":"ContainerStarted","Data":"e4a5f06440cefc71e779e0e7a1886b7e57ce403bd1eb2c0f5f82e90deebf20a6"}
Nov 03 14:29:32.054458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.054328    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt" event={"ID":"770bc1b5-ca92-42a8-874f-da444a26c0f4","Type":"ContainerStarted","Data":"aa7964494ed01b455c41041b53b4b10870a6ee22155c57d12910a8afe4195ce6"}
Nov 03 14:29:32.055784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.055722    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:32.119516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.119464    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt" podStartSLOduration=2.119445558 podStartE2EDuration="2.119445558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:32.116373363 +0000 UTC m=+1050.166404547" watchObservedRunningTime="2026-11-03 14:29:32.119445558 +0000 UTC m=+1050.169476736"
Nov 03 14:29:32.380646 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:29:32.380612    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30f0686_e3ff_418f_834c_f6cab1435b44.slice/crio-5a02f736e5df0872fa64cbdbe340c44fa7d5d74d8a5d615c4b243dd2c4edec55 WatchSource:0}: Error finding container 5a02f736e5df0872fa64cbdbe340c44fa7d5d74d8a5d615c4b243dd2c4edec55: Status 404 returned error can't find the container with id 5a02f736e5df0872fa64cbdbe340c44fa7d5d74d8a5d615c4b243dd2c4edec55
Nov 03 14:29:32.381222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.381186    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"]
Nov 03 14:29:32.996418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:32.996322    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:33.063087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:33.063052    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl" event={"ID":"e30f0686-e3ff-418f-834c-f6cab1435b44","Type":"ContainerStarted","Data":"6cb734cf29238d54a65e43a28a773c2f65f8feffdae638b5f77f0ec3905b2ddf"}
Nov 03 14:29:33.063289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:33.063112    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl" event={"ID":"e30f0686-e3ff-418f-834c-f6cab1435b44","Type":"ContainerStarted","Data":"5a02f736e5df0872fa64cbdbe340c44fa7d5d74d8a5d615c4b243dd2c4edec55"}
Nov 03 14:29:33.113333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:33.106579    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl" podStartSLOduration=2.106558037 podStartE2EDuration="2.106558037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:33.103623443 +0000 UTC m=+1051.153654626" watchObservedRunningTime="2026-11-03 14:29:33.106558037 +0000 UTC m=+1051.156589221"
Nov 03 14:29:34.071043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:34.071011    6481 generic.go:356] "Generic (PLEG): container finished" podID="e30f0686-e3ff-418f-834c-f6cab1435b44" containerID="6cb734cf29238d54a65e43a28a773c2f65f8feffdae638b5f77f0ec3905b2ddf" exitCode=0
Nov 03 14:29:34.071446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:34.071092    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl" event={"ID":"e30f0686-e3ff-418f-834c-f6cab1435b44","Type":"ContainerDied","Data":"6cb734cf29238d54a65e43a28a773c2f65f8feffdae638b5f77f0ec3905b2ddf"}
Nov 03 14:29:34.993110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:34.993078    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:35.229679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.229654    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:35.307478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.307394    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss\" (UniqueName: \"kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss\") pod \"e30f0686-e3ff-418f-834c-f6cab1435b44\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") "
Nov 03 14:29:35.307636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.307500    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca\") pod \"e30f0686-e3ff-418f-834c-f6cab1435b44\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") "
Nov 03 14:29:35.307699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.307665    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca\") pod \"e30f0686-e3ff-418f-834c-f6cab1435b44\" (UID: \"e30f0686-e3ff-418f-834c-f6cab1435b44\") "
Nov 03 14:29:35.308028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.307997    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca" pod "e30f0686-e3ff-418f-834c-f6cab1435b44" (UID: "e30f0686-e3ff-418f-834c-f6cab1435b44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:35.308323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.308300    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:35.309749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.309722    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca" pod "e30f0686-e3ff-418f-834c-f6cab1435b44" (UID: "e30f0686-e3ff-418f-834c-f6cab1435b44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:29:35.309749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.309735    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss" pod "e30f0686-e3ff-418f-834c-f6cab1435b44" (UID: "e30f0686-e3ff-418f-834c-f6cab1435b44"). InnerVolumeSpecName "kube-api-access-pstss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:35.409028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.408998    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstss\" (UniqueName: \"kubernetes.io/projected/e30f0686-e3ff-418f-834c-f6cab1435b44-kube-api-access-pstss\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:35.409028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:35.409026    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e30f0686-e3ff-418f-834c-f6cab1435b44-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:36.087046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:36.087020    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"
Nov 03 14:29:36.087190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:36.087059    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl" event={"ID":"e30f0686-e3ff-418f-834c-f6cab1435b44","Type":"ContainerDied","Data":"5a02f736e5df0872fa64cbdbe340c44fa7d5d74d8a5d615c4b243dd2c4edec55"}
Nov 03 14:29:36.087190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:36.087111    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a02f736e5df0872fa64cbdbe340c44fa7d5d74d8a5d615c4b243dd2c4edec55"
Nov 03 14:29:36.976857 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:36.976826    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:38.978130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:38.978090    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:39.045356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:39.045312    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"]
Nov 03 14:29:39.054680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:39.054651    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883e8847-d8pl6"]
Nov 03 14:29:40.067503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:40.067470    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd64381-bb15-40c4-a9dc-a99760b00cff" path="/var/lib/kubelet/pods/0fd64381-bb15-40c4-a9dc-a99760b00cff/volumes"
Nov 03 14:29:40.712354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:40.712320    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:41.422406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:41.422375    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:41.633200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:41.633170    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:41.703159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:41.703076    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:29:41.844866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:41.844830    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:42.434750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:42.434713    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:42.664922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:42.664888    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:42.885320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:42.885242    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:43.247978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:43.247952    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:43.469372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:43.469341    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:43.686860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:43.686835    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:44.042790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:44.042716    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"]
Nov 03 14:29:44.053920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:44.053894    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19f4127c-9zx9n"]
Nov 03 14:29:44.068717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:44.068686    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9005fd-a7bc-404d-9bb3-db7cb9dea027" path="/var/lib/kubelet/pods/6d9005fd-a7bc-404d-9bb3-db7cb9dea027/volumes"
Nov 03 14:29:44.074735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:44.074714    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:29:44.153977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:44.153942    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"]
Nov 03 14:29:44.191137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:44.191108    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:46.479252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.476259    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:46.534732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.534698    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"]
Nov 03 14:29:46.536011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.535974    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e30f0686-e3ff-418f-834c-f6cab1435b44" containerName="revoke-keys"
Nov 03 14:29:46.536011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.536001    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30f0686-e3ff-418f-834c-f6cab1435b44" containerName="revoke-keys"
Nov 03 14:29:46.536214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.536202    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30f0686-e3ff-418f-834c-f6cab1435b44" containerName="revoke-keys"
Nov 03 14:29:46.547199 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.547175    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.557063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.556480    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"]
Nov 03 14:29:46.566214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.566189    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:29:46.573366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.573342    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw22n\" (UniqueName: \"kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.573491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.573381    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.573541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.573525    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.577174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.577144    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:29:46.577174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.577164    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:29:46.577308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.577208    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:29:46.674389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.674358    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.674574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.674503    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw22n\" (UniqueName: \"kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.674574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.674525    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.675135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.675113    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.676749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.676725    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.688968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.688940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:46.696977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.696956    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw22n\" (UniqueName: \"kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n\") pod \"maas-api-revoke-sub-0da1c2b9-rh9t8\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:46.872498 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:46.872424    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:47.180446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:47.180415    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"]
Nov 03 14:29:47.181021 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:29:47.180997    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d15f2c_0eaf_446f_bd9a_24cabca7378a.slice/crio-dac0f03bdcc6645f6af04b4d778d495a00463dfe4dae8f55ac773b4d62dd37bb WatchSource:0}: Error finding container dac0f03bdcc6645f6af04b4d778d495a00463dfe4dae8f55ac773b4d62dd37bb: Status 404 returned error can't find the container with id dac0f03bdcc6645f6af04b4d778d495a00463dfe4dae8f55ac773b4d62dd37bb
Nov 03 14:29:47.221060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:47.221033    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8" event={"ID":"d7d15f2c-0eaf-446f-bd9a-24cabca7378a","Type":"ContainerStarted","Data":"dac0f03bdcc6645f6af04b4d778d495a00463dfe4dae8f55ac773b4d62dd37bb"}
Nov 03 14:29:47.489250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:47.489150    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:47.730187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:47.730039    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:48.230071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:48.230031    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8" event={"ID":"d7d15f2c-0eaf-446f-bd9a-24cabca7378a","Type":"ContainerStarted","Data":"15a850bc8e10b63e4b8bd4b0571f5521a321dfe170ca53790eed788d07ea0599"}
Nov 03 14:29:48.277716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:48.277671    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8" podStartSLOduration=2.27765795 podStartE2EDuration="2.27765795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:48.275052132 +0000 UTC m=+1066.325083315" watchObservedRunningTime="2026-11-03 14:29:48.27765795 +0000 UTC m=+1066.327689132"
Nov 03 14:29:48.951662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:48.951626    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:49.154603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.154565    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b" podUID="7dfb3ee4-75e8-4a9f-bfd0-721429024e02" containerName="payload-processing" containerID="cri-o://3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781" gracePeriod=25
Nov 03 14:29:49.248382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.248298    6481 generic.go:356] "Generic (PLEG): container finished" podID="d7d15f2c-0eaf-446f-bd9a-24cabca7378a" containerID="15a850bc8e10b63e4b8bd4b0571f5521a321dfe170ca53790eed788d07ea0599" exitCode=0
Nov 03 14:29:49.248510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.248381    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8" event={"ID":"d7d15f2c-0eaf-446f-bd9a-24cabca7378a","Type":"ContainerDied","Data":"15a850bc8e10b63e4b8bd4b0571f5521a321dfe170ca53790eed788d07ea0599"}
Nov 03 14:29:49.445908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.445881    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:49.522978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.522892    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume\") pod \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") "
Nov 03 14:29:49.522978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.522931    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr\") pod \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") "
Nov 03 14:29:49.523159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.523136    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials\") pod \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") "
Nov 03 14:29:49.523196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.523166    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume\") pod \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\" (UID: \"7dfb3ee4-75e8-4a9f-bfd0-721429024e02\") "
Nov 03 14:29:49.523385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.523360    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume" pod "7dfb3ee4-75e8-4a9f-bfd0-721429024e02" (UID: "7dfb3ee4-75e8-4a9f-bfd0-721429024e02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:49.523724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.523700    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume" pod "7dfb3ee4-75e8-4a9f-bfd0-721429024e02" (UID: "7dfb3ee4-75e8-4a9f-bfd0-721429024e02"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:49.526738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.523888    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:49.526738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.523908    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:49.526738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.525459    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials" pod "7dfb3ee4-75e8-4a9f-bfd0-721429024e02" (UID: "7dfb3ee4-75e8-4a9f-bfd0-721429024e02"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:49.526738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.526030    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr" pod "7dfb3ee4-75e8-4a9f-bfd0-721429024e02" (UID: "7dfb3ee4-75e8-4a9f-bfd0-721429024e02"). InnerVolumeSpecName "kube-api-access-l77rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:49.625648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.625619    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:49.625648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:49.625646    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l77rr\" (UniqueName: \"kubernetes.io/projected/7dfb3ee4-75e8-4a9f-bfd0-721429024e02-kube-api-access-l77rr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:50.256657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.256622    6481 generic.go:356] "Generic (PLEG): container finished" podID="7dfb3ee4-75e8-4a9f-bfd0-721429024e02" containerID="3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781" exitCode=0
Nov 03 14:29:50.257193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.256683    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"
Nov 03 14:29:50.257193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.256707    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b" event={"ID":"7dfb3ee4-75e8-4a9f-bfd0-721429024e02","Type":"ContainerDied","Data":"3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781"}
Nov 03 14:29:50.257193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.256772    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b" event={"ID":"7dfb3ee4-75e8-4a9f-bfd0-721429024e02","Type":"ContainerDied","Data":"2e8b9620a388952f6b7dc8a9aa023657769c4ea0d69f03ccc14f884f07fdff37"}
Nov 03 14:29:50.257193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.256802    6481 scope.go:122] "RemoveContainer" containerID="3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781"
Nov 03 14:29:50.269867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.269845    6481 scope.go:122] "RemoveContainer" containerID="3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781"
Nov 03 14:29:50.270155 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:29:50.270132    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781\": container with ID starting with 3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781 not found: ID does not exist" containerID="3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781"
Nov 03 14:29:50.270246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.270163    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781"} err="failed to get container status \"3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781\": rpc error: code = NotFound desc = could not find container \"3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781\": container with ID starting with 3b4060ab9cab45672286b64c51b47309a8d53a54c95f4f12299ddba055ac6781 not found: ID does not exist"
Nov 03 14:29:50.312222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.312185    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"]
Nov 03 14:29:50.326581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.326514    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b54z5l5b"]
Nov 03 14:29:50.413471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.413447    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:50.437523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.437500    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca\") pod \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") "
Nov 03 14:29:50.437640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.437618    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca\") pod \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") "
Nov 03 14:29:50.437754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.437735    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n\" (UniqueName: \"kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n\") pod \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\" (UID: \"d7d15f2c-0eaf-446f-bd9a-24cabca7378a\") "
Nov 03 14:29:50.437844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.437812    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca" pod "d7d15f2c-0eaf-446f-bd9a-24cabca7378a" (UID: "d7d15f2c-0eaf-446f-bd9a-24cabca7378a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:29:50.438201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.438183    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:50.439710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.439690    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca" pod "d7d15f2c-0eaf-446f-bd9a-24cabca7378a" (UID: "d7d15f2c-0eaf-446f-bd9a-24cabca7378a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:29:50.439812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.439791    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n" pod "d7d15f2c-0eaf-446f-bd9a-24cabca7378a" (UID: "d7d15f2c-0eaf-446f-bd9a-24cabca7378a"). InnerVolumeSpecName "kube-api-access-zw22n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:29:50.539875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.539845    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:50.539875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:50.539871    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw22n\" (UniqueName: \"kubernetes.io/projected/d7d15f2c-0eaf-446f-bd9a-24cabca7378a-kube-api-access-zw22n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:29:51.110045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:51.109968    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:51.266388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:51.266359    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"
Nov 03 14:29:51.266861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:51.266386    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8" event={"ID":"d7d15f2c-0eaf-446f-bd9a-24cabca7378a","Type":"ContainerDied","Data":"dac0f03bdcc6645f6af04b4d778d495a00463dfe4dae8f55ac773b4d62dd37bb"}
Nov 03 14:29:51.266861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:51.266447    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dac0f03bdcc6645f6af04b4d778d495a00463dfe4dae8f55ac773b4d62dd37bb"
Nov 03 14:29:51.342977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:51.342951    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:51.997811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:51.997764    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"]
Nov 03 14:29:52.077056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:52.077023    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dfb3ee4-75e8-4a9f-bfd0-721429024e02" path="/var/lib/kubelet/pods/7dfb3ee4-75e8-4a9f-bfd0-721429024e02/volumes"
Nov 03 14:29:52.152782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:52.152755    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:53.738258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:53.733993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:54.134301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:54.133553    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"]
Nov 03 14:29:54.184341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:54.184306    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f46ed72-jrzjq"]
Nov 03 14:29:55.898222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:55.898190    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:56.071496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:56.071458    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2651061b-aedc-49a6-a2c0-aa645a74a7c6" path="/var/lib/kubelet/pods/2651061b-aedc-49a6-a2c0-aa645a74a7c6/volumes"
Nov 03 14:29:57.905322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:57.905281    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:57.999890 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:29:57.999829    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:29:58.116249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:58.116183    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:58.552104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:58.552069    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:29:59.424477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.424447    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"]
Nov 03 14:29:59.425022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.425006    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d7d15f2c-0eaf-446f-bd9a-24cabca7378a" containerName="revoke-keys"
Nov 03 14:29:59.425065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.425023    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d15f2c-0eaf-446f-bd9a-24cabca7378a" containerName="revoke-keys"
Nov 03 14:29:59.425065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.425041    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7dfb3ee4-75e8-4a9f-bfd0-721429024e02" containerName="payload-processing"
Nov 03 14:29:59.425065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.425046    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dfb3ee4-75e8-4a9f-bfd0-721429024e02" containerName="payload-processing"
Nov 03 14:29:59.425189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.425180    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d15f2c-0eaf-446f-bd9a-24cabca7378a" containerName="revoke-keys"
Nov 03 14:29:59.425225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.425192    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dfb3ee4-75e8-4a9f-bfd0-721429024e02" containerName="payload-processing"
Nov 03 14:29:59.430486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.430467    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.450678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.450652    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"]
Nov 03 14:29:59.571214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.571178    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.571408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.571323    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.571491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.571460    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.571529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.571502    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhqrz\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.672675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.672639    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.672675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.672681    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhqrz\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.672891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.672813    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.672945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.672928    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.673446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.673420    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.673564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.673514    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.675564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.675521    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.694343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.694319    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhqrz\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.744397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.744377    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:29:59.957133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:29:59.957051    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:00.060693 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:00.060660    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bdf93f_f1fa_4870_9d37_b56f812cfd04.slice/crio-acf5d1aa8e596bf5662f854e9dc195575dfa1c9fa441af98a58189219acf74e5 WatchSource:0}: Error finding container acf5d1aa8e596bf5662f854e9dc195575dfa1c9fa441af98a58189219acf74e5: Status 404 returned error can't find the container with id acf5d1aa8e596bf5662f854e9dc195575dfa1c9fa441af98a58189219acf74e5
Nov 03 14:30:00.068331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.068306    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"]
Nov 03 14:30:00.175744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.175699    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"]
Nov 03 14:30:00.180620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.180600    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"]
Nov 03 14:30:00.180741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.180727    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.185384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.185354    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:00.186581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.186562    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"]
Nov 03 14:30:00.186707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.186691    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.191825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.191806    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.202170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.202149    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:30:00.215208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.215148    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:30:00.215383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.215266    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:30:00.215383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.215337    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"]
Nov 03 14:30:00.221190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.219048    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"]
Nov 03 14:30:00.229138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.228422    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"]
Nov 03 14:30:00.253950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.253014    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"]
Nov 03 14:30:00.260447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.260425    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.266377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.266351    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"]
Nov 03 14:30:00.320465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.320436    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4scn\" (UniqueName: \"kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.320600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.320482    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.320639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.320603    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tsc\" (UniqueName: \"kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc\") pod \"maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.320686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.320672    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.320728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.320696    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6sg4\" (UniqueName: \"kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.320762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.320737    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.422437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.422398    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4scn\" (UniqueName: \"kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.422635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.422467    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.422635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.422565    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tsc\" (UniqueName: \"kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc\") pod \"maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.422787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.422647    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.422787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.422676    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6sg4\" (UniqueName: \"kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.422787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.422717    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.423075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.423049    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.423140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.423123    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtvjl\" (UniqueName: \"kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.423434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.423403    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.423554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.423458    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.423788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.423761    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.443542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.443516    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4scn\" (UniqueName: \"kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.444089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.444067    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tsc\" (UniqueName: \"kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc\") pod \"maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.444272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.444258    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6sg4\" (UniqueName: \"kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.498545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.498462    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:00.505136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.505107    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l" event={"ID":"a4bdf93f-f1fa-4870-9d37-b56f812cfd04","Type":"ContainerStarted","Data":"a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8"}
Nov 03 14:30:00.505285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.505167    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l" event={"ID":"a4bdf93f-f1fa-4870-9d37-b56f812cfd04","Type":"ContainerStarted","Data":"acf5d1aa8e596bf5662f854e9dc195575dfa1c9fa441af98a58189219acf74e5"}
Nov 03 14:30:00.505285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.505199    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:30:00.524902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.524869    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.525024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.524958    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtvjl\" (UniqueName: \"kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.526305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.526215    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.549562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.549503    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l" podStartSLOduration=1.549484975 podStartE2EDuration="1.549484975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:00.549173514 +0000 UTC m=+1078.599204709" watchObservedRunningTime="2026-11-03 14:30:00.549484975 +0000 UTC m=+1078.599516158"
Nov 03 14:30:00.552691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.552669    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtvjl\" (UniqueName: \"kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl\") pod \"maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.556434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.556416    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:00.568018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.567045    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:00.580005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.576504    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:00.824416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.824392    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"]
Nov 03 14:30:00.824957 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:00.824936    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a364cb_17b8_4e41_a2aa_0eddf12f741a.slice/crio-99b5dc240ed0993f8a14edee2a7046b99b4319d1e46ce0bb4ba36f8c280cf67a WatchSource:0}: Error finding container 99b5dc240ed0993f8a14edee2a7046b99b4319d1e46ce0bb4ba36f8c280cf67a: Status 404 returned error can't find the container with id 99b5dc240ed0993f8a14edee2a7046b99b4319d1e46ce0bb4ba36f8c280cf67a
Nov 03 14:30:00.919397 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:00.919368    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2690fc59_61ca_4197_894b_8a4c2754f1f6.slice/crio-0ea1e99312ac67cb46c5223ff8b6a768e4d7764126ea601e7b7f536297b25ed0 WatchSource:0}: Error finding container 0ea1e99312ac67cb46c5223ff8b6a768e4d7764126ea601e7b7f536297b25ed0: Status 404 returned error can't find the container with id 0ea1e99312ac67cb46c5223ff8b6a768e4d7764126ea601e7b7f536297b25ed0
Nov 03 14:30:00.920841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.920823    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"]
Nov 03 14:30:00.922638 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:00.922565    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167425fa_3244_4a80_8630_80b0d687ce4b.slice/crio-a55bfc5896d35ed78149bfb7611456d497fe6b0dc2058aa42afb54008ed3d16d WatchSource:0}: Error finding container a55bfc5896d35ed78149bfb7611456d497fe6b0dc2058aa42afb54008ed3d16d: Status 404 returned error can't find the container with id a55bfc5896d35ed78149bfb7611456d497fe6b0dc2058aa42afb54008ed3d16d
Nov 03 14:30:00.926068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.926050    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"]
Nov 03 14:30:00.938308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.938286    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"]
Nov 03 14:30:00.943212 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:00.943182    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00777ec_effb_4bbe_a9e0_0a3dea4c2c78.slice/crio-0f301bda14448051cf1028abb6e1316b9b54ddeea41414fc314c9b5689c22266 WatchSource:0}: Error finding container 0f301bda14448051cf1028abb6e1316b9b54ddeea41414fc314c9b5689c22266: Status 404 returned error can't find the container with id 0f301bda14448051cf1028abb6e1316b9b54ddeea41414fc314c9b5689c22266
Nov 03 14:30:00.990001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.989975    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"]
Nov 03 14:30:00.995600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:00.995582    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.012014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.011993    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:30:01.014738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.014710    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"]
Nov 03 14:30:01.134515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.134490    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.134628 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.134611    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs6lw\" (UniqueName: \"kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.134682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.134671    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.236066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.236033    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.236200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.236183    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.236321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.236304    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs6lw\" (UniqueName: \"kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.236866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.236846    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.238632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.238614    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.257561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.257538    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs6lw\" (UniqueName: \"kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw\") pod \"maas-api-revoke-sub-848771fc-mbcsp\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.311774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.311750    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:01.514325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.514285    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerStarted","Data":"e3dbf6076ba2a9de98ef345e854b97d3a1a1bdb377d3ce2e4b1d854333a8a864"}
Nov 03 14:30:01.514715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.514347    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerStarted","Data":"0f301bda14448051cf1028abb6e1316b9b54ddeea41414fc314c9b5689c22266"}
Nov 03 14:30:01.515975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.515946    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerStarted","Data":"cc9cf9393b80907e7ed6b04b52ce2c1378f5dc93eb3db175949a73ad40adf9bd"}
Nov 03 14:30:01.516090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.515996    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerStarted","Data":"0ea1e99312ac67cb46c5223ff8b6a768e4d7764126ea601e7b7f536297b25ed0"}
Nov 03 14:30:01.517613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.517587    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerStarted","Data":"c55887bbcc9c6463bf0c8f5af9108a2fb7d14cb547b4222ea522cc5e582d8434"}
Nov 03 14:30:01.517739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.517637    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerStarted","Data":"99b5dc240ed0993f8a14edee2a7046b99b4319d1e46ce0bb4ba36f8c280cf67a"}
Nov 03 14:30:01.519141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.519121    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerStarted","Data":"7a061426d5105bc6c784eeb7334f02c261017b7597b2ea7f26d03fe40cee5d40"}
Nov 03 14:30:01.519348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.519158    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerStarted","Data":"a55bfc5896d35ed78149bfb7611456d497fe6b0dc2058aa42afb54008ed3d16d"}
Nov 03 14:30:01.562991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.558146    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" podStartSLOduration=1.5581285710000001 podStartE2EDuration="1.558128571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.556970889 +0000 UTC m=+1079.607002069" watchObservedRunningTime="2026-11-03 14:30:01.558128571 +0000 UTC m=+1079.608159756"
Nov 03 14:30:01.600151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.600100    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" podStartSLOduration=1.600086605 podStartE2EDuration="1.600086605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.596680539 +0000 UTC m=+1079.646711723" watchObservedRunningTime="2026-11-03 14:30:01.600086605 +0000 UTC m=+1079.650117787"
Nov 03 14:30:01.616407 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:01.616376    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5cd5f80_9d5a_4779_9f71_c874dbf99c51.slice/crio-deb187e633cc6bd87315eaa9baf163e8d74d4da3408a108ebf75755308724f27 WatchSource:0}: Error finding container deb187e633cc6bd87315eaa9baf163e8d74d4da3408a108ebf75755308724f27: Status 404 returned error can't find the container with id deb187e633cc6bd87315eaa9baf163e8d74d4da3408a108ebf75755308724f27
Nov 03 14:30:01.617119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.617080    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"]
Nov 03 14:30:01.639260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.639197    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" podStartSLOduration=1.639182802 podStartE2EDuration="1.639182802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.637912861 +0000 UTC m=+1079.687944044" watchObservedRunningTime="2026-11-03 14:30:01.639182802 +0000 UTC m=+1079.689213984"
Nov 03 14:30:01.678872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:01.678826    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" podStartSLOduration=1.678813065 podStartE2EDuration="1.678813065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.677371996 +0000 UTC m=+1079.727403178" watchObservedRunningTime="2026-11-03 14:30:01.678813065 +0000 UTC m=+1079.728844297"
Nov 03 14:30:02.016013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.015980    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:02.528480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.528446    6481 generic.go:356] "Generic (PLEG): container finished" podID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerID="c55887bbcc9c6463bf0c8f5af9108a2fb7d14cb547b4222ea522cc5e582d8434" exitCode=60
Nov 03 14:30:02.528894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.528528    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerDied","Data":"c55887bbcc9c6463bf0c8f5af9108a2fb7d14cb547b4222ea522cc5e582d8434"}
Nov 03 14:30:02.528894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.528764    6481 scope.go:122] "RemoveContainer" containerID="c55887bbcc9c6463bf0c8f5af9108a2fb7d14cb547b4222ea522cc5e582d8434"
Nov 03 14:30:02.530470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.530371    6481 generic.go:356] "Generic (PLEG): container finished" podID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerID="e3dbf6076ba2a9de98ef345e854b97d3a1a1bdb377d3ce2e4b1d854333a8a864" exitCode=60
Nov 03 14:30:02.530563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.530487    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerDied","Data":"e3dbf6076ba2a9de98ef345e854b97d3a1a1bdb377d3ce2e4b1d854333a8a864"}
Nov 03 14:30:02.530701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.530686    6481 scope.go:122] "RemoveContainer" containerID="e3dbf6076ba2a9de98ef345e854b97d3a1a1bdb377d3ce2e4b1d854333a8a864"
Nov 03 14:30:02.532157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.532140    6481 generic.go:356] "Generic (PLEG): container finished" podID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerID="cc9cf9393b80907e7ed6b04b52ce2c1378f5dc93eb3db175949a73ad40adf9bd" exitCode=60
Nov 03 14:30:02.532255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.532225    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerDied","Data":"cc9cf9393b80907e7ed6b04b52ce2c1378f5dc93eb3db175949a73ad40adf9bd"}
Nov 03 14:30:02.532495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.532403    6481 scope.go:122] "RemoveContainer" containerID="cc9cf9393b80907e7ed6b04b52ce2c1378f5dc93eb3db175949a73ad40adf9bd"
Nov 03 14:30:02.533968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.533947    6481 generic.go:356] "Generic (PLEG): container finished" podID="167425fa-3244-4a80-8630-80b0d687ce4b" containerID="7a061426d5105bc6c784eeb7334f02c261017b7597b2ea7f26d03fe40cee5d40" exitCode=60
Nov 03 14:30:02.534034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.534000    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerDied","Data":"7a061426d5105bc6c784eeb7334f02c261017b7597b2ea7f26d03fe40cee5d40"}
Nov 03 14:30:02.534202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.534188    6481 scope.go:122] "RemoveContainer" containerID="7a061426d5105bc6c784eeb7334f02c261017b7597b2ea7f26d03fe40cee5d40"
Nov 03 14:30:02.539102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.538286    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp" event={"ID":"d5cd5f80-9d5a-4779-9f71-c874dbf99c51","Type":"ContainerStarted","Data":"e350848b9673f96b52f67398f74903aaee3e5770b7cf9b2b3e4ba2c6cd52507f"}
Nov 03 14:30:02.539102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.538357    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp" event={"ID":"d5cd5f80-9d5a-4779-9f71-c874dbf99c51","Type":"ContainerStarted","Data":"deb187e633cc6bd87315eaa9baf163e8d74d4da3408a108ebf75755308724f27"}
Nov 03 14:30:02.749763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:02.749712    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp" podStartSLOduration=2.74969958 podStartE2EDuration="2.74969958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:02.745833885 +0000 UTC m=+1080.795865050" watchObservedRunningTime="2026-11-03 14:30:02.74969958 +0000 UTC m=+1080.799730794"
Nov 03 14:30:03.148552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.148477    6481 scope.go:122] "RemoveContainer" containerID="3ab50dfb1c0ebb9030707aca15501651acdbc274a8a46a13422af6f293da9761"
Nov 03 14:30:03.159253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.159217    6481 scope.go:122] "RemoveContainer" containerID="d7a5890b5d0dce4bc5435aaddcf5c80b256f32d555afd4f77540128c1fbb98c3"
Nov 03 14:30:03.168461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.168441    6481 scope.go:122] "RemoveContainer" containerID="96bc72f7acc1d84c5716525ce133995be4cea3056c9e74ee3b08f706c1a504b1"
Nov 03 14:30:03.548724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.548691    6481 generic.go:356] "Generic (PLEG): container finished" podID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerID="6b14976fcd1351b8d6584d1eab2f8706664bebcad6f9acd00d03e82283712c0d" exitCode=60
Nov 03 14:30:03.549103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.548770    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerDied","Data":"6b14976fcd1351b8d6584d1eab2f8706664bebcad6f9acd00d03e82283712c0d"}
Nov 03 14:30:03.549103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.548838    6481 scope.go:122] "RemoveContainer" containerID="cc9cf9393b80907e7ed6b04b52ce2c1378f5dc93eb3db175949a73ad40adf9bd"
Nov 03 14:30:03.549221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.549153    6481 scope.go:122] "RemoveContainer" containerID="6b14976fcd1351b8d6584d1eab2f8706664bebcad6f9acd00d03e82283712c0d"
Nov 03 14:30:03.549555 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:03.549531    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9_odh-ai-gateway-infra(2690fc59-61ca-4197-894b-8a4c2754f1f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6"
Nov 03 14:30:03.550744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.550716    6481 generic.go:356] "Generic (PLEG): container finished" podID="167425fa-3244-4a80-8630-80b0d687ce4b" containerID="3c4ad5a25d814a9ebd581ebadfc8057575b63e50ab7d710ee7dac1a22ec3630b" exitCode=60
Nov 03 14:30:03.550848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.550793    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerDied","Data":"3c4ad5a25d814a9ebd581ebadfc8057575b63e50ab7d710ee7dac1a22ec3630b"}
Nov 03 14:30:03.551087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.551072    6481 scope.go:122] "RemoveContainer" containerID="3c4ad5a25d814a9ebd581ebadfc8057575b63e50ab7d710ee7dac1a22ec3630b"
Nov 03 14:30:03.551439 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:03.551415    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd_odh-ai-gateway-infra(167425fa-3244-4a80-8630-80b0d687ce4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" podUID="167425fa-3244-4a80-8630-80b0d687ce4b"
Nov 03 14:30:03.552472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.552453    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5cd5f80-9d5a-4779-9f71-c874dbf99c51" containerID="e350848b9673f96b52f67398f74903aaee3e5770b7cf9b2b3e4ba2c6cd52507f" exitCode=0
Nov 03 14:30:03.552566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.552534    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp" event={"ID":"d5cd5f80-9d5a-4779-9f71-c874dbf99c51","Type":"ContainerDied","Data":"e350848b9673f96b52f67398f74903aaee3e5770b7cf9b2b3e4ba2c6cd52507f"}
Nov 03 14:30:03.554460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.554435    6481 generic.go:356] "Generic (PLEG): container finished" podID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerID="9c061895dfaea69225da7df343c488cc42b7a4d74fb793f791d36417e34a297a" exitCode=60
Nov 03 14:30:03.554570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.554510    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerDied","Data":"9c061895dfaea69225da7df343c488cc42b7a4d74fb793f791d36417e34a297a"}
Nov 03 14:30:03.554807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.554788    6481 scope.go:122] "RemoveContainer" containerID="9c061895dfaea69225da7df343c488cc42b7a4d74fb793f791d36417e34a297a"
Nov 03 14:30:03.555108 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:03.555084    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l_odh-ai-gateway-infra(27a364cb-17b8-4e41-a2aa-0eddf12f741a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a"
Nov 03 14:30:03.556461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.556439    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerStarted","Data":"46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2"}
Nov 03 14:30:03.744617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.744593    6481 scope.go:122] "RemoveContainer" containerID="7a061426d5105bc6c784eeb7334f02c261017b7597b2ea7f26d03fe40cee5d40"
Nov 03 14:30:03.758115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:03.758094    6481 scope.go:122] "RemoveContainer" containerID="c55887bbcc9c6463bf0c8f5af9108a2fb7d14cb547b4222ea522cc5e582d8434"
Nov 03 14:30:04.045018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.044948    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"]
Nov 03 14:30:04.051927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.051899    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af6f73ee-c26q8"]
Nov 03 14:30:04.052966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.052940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:04.069906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.069882    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf920a72-cd34-40d7-b34d-defdecf3acf0" path="/var/lib/kubelet/pods/cf920a72-cd34-40d7-b34d-defdecf3acf0/volumes"
Nov 03 14:30:04.339847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.339752    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:04.565072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.565045    6481 scope.go:122] "RemoveContainer" containerID="6b14976fcd1351b8d6584d1eab2f8706664bebcad6f9acd00d03e82283712c0d"
Nov 03 14:30:04.565549 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:04.565334    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9_odh-ai-gateway-infra(2690fc59-61ca-4197-894b-8a4c2754f1f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6"
Nov 03 14:30:04.566839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.566818    6481 scope.go:122] "RemoveContainer" containerID="3c4ad5a25d814a9ebd581ebadfc8057575b63e50ab7d710ee7dac1a22ec3630b"
Nov 03 14:30:04.567096 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:04.567072    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd_odh-ai-gateway-infra(167425fa-3244-4a80-8630-80b0d687ce4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" podUID="167425fa-3244-4a80-8630-80b0d687ce4b"
Nov 03 14:30:04.568396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.568377    6481 scope.go:122] "RemoveContainer" containerID="9c061895dfaea69225da7df343c488cc42b7a4d74fb793f791d36417e34a297a"
Nov 03 14:30:04.568632 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:04.568612    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l_odh-ai-gateway-infra(27a364cb-17b8-4e41-a2aa-0eddf12f741a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a"
Nov 03 14:30:04.569888 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.569863    6481 generic.go:356] "Generic (PLEG): container finished" podID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerID="46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2" exitCode=60
Nov 03 14:30:04.569980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.569923    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerDied","Data":"46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2"}
Nov 03 14:30:04.570023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.569984    6481 scope.go:122] "RemoveContainer" containerID="e3dbf6076ba2a9de98ef345e854b97d3a1a1bdb377d3ce2e4b1d854333a8a864"
Nov 03 14:30:04.570127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.570107    6481 scope.go:122] "RemoveContainer" containerID="46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2"
Nov 03 14:30:04.570622 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:04.570363    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh_odh-ai-gateway-infra(b00777ec-effb-4bbe-a9e0-0a3dea4c2c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78"
Nov 03 14:30:04.713729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.713704    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:04.883345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.883253    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw\" (UniqueName: \"kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw\") pod \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") "
Nov 03 14:30:04.883345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.883301    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca\") pod \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") "
Nov 03 14:30:04.883544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.883525    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca\") pod \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\" (UID: \"d5cd5f80-9d5a-4779-9f71-c874dbf99c51\") "
Nov 03 14:30:04.883691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.883665    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca" pod "d5cd5f80-9d5a-4779-9f71-c874dbf99c51" (UID: "d5cd5f80-9d5a-4779-9f71-c874dbf99c51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:04.883999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.883978    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:04.885572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.885545    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca" pod "d5cd5f80-9d5a-4779-9f71-c874dbf99c51" (UID: "d5cd5f80-9d5a-4779-9f71-c874dbf99c51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:30:04.885664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.885592    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw" pod "d5cd5f80-9d5a-4779-9f71-c874dbf99c51" (UID: "d5cd5f80-9d5a-4779-9f71-c874dbf99c51"). InnerVolumeSpecName "kube-api-access-zs6lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:04.985026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.984983    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs6lw\" (UniqueName: \"kubernetes.io/projected/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-kube-api-access-zs6lw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:04.985026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:04.985022    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5cd5f80-9d5a-4779-9f71-c874dbf99c51-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:05.006360 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:05.006295    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:30:05.639133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:05.626446    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp" event={"ID":"d5cd5f80-9d5a-4779-9f71-c874dbf99c51","Type":"ContainerDied","Data":"deb187e633cc6bd87315eaa9baf163e8d74d4da3408a108ebf75755308724f27"}
Nov 03 14:30:05.639133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:05.626560    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb187e633cc6bd87315eaa9baf163e8d74d4da3408a108ebf75755308724f27"
Nov 03 14:30:05.639133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:05.626632    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"
Nov 03 14:30:05.648251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:05.646545    6481 scope.go:122] "RemoveContainer" containerID="46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2"
Nov 03 14:30:05.648251 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:05.646918    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh_odh-ai-gateway-infra(b00777ec-effb-4bbe-a9e0-0a3dea4c2c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78"
Nov 03 14:30:06.377907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:06.377869    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:07.024420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:07.024381    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" containerID="cri-o://c33651932a7bacaf225c6ede35b7d47787b8485a4fc99118db6c251048b5be91" gracePeriod=45
Nov 03 14:30:07.619254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:07.618687    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:07.723255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:07.714015    6481 generic.go:356] "Generic (PLEG): container finished" podID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerID="c33651932a7bacaf225c6ede35b7d47787b8485a4fc99118db6c251048b5be91" exitCode=0
Nov 03 14:30:07.723255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:07.714223    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" event={"ID":"bdd9d7fb-c03c-4814-9aee-6f3927d3e559","Type":"ContainerDied","Data":"c33651932a7bacaf225c6ede35b7d47787b8485a4fc99118db6c251048b5be91"}
Nov 03 14:30:07.832686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:07.832587    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": dial tcp 10.133.0.89:8000: connect: connection refused"
Nov 03 14:30:08.192517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.192490    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:30:08.320991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.320959    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir\") pod \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") "
Nov 03 14:30:08.321187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321085    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home\") pod \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") "
Nov 03 14:30:08.321279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321261    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs\") pod \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") "
Nov 03 14:30:08.321349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321323    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp\" (UniqueName: \"kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp\") pod \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") "
Nov 03 14:30:08.321349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321256    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir" pod "bdd9d7fb-c03c-4814-9aee-6f3927d3e559" (UID: "bdd9d7fb-c03c-4814-9aee-6f3927d3e559"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:30:08.321452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321348    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home" pod "bdd9d7fb-c03c-4814-9aee-6f3927d3e559" (UID: "bdd9d7fb-c03c-4814-9aee-6f3927d3e559"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:30:08.321452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321391    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache\") pod \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") "
Nov 03 14:30:08.321452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321430    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm\") pod \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\" (UID: \"bdd9d7fb-c03c-4814-9aee-6f3927d3e559\") "
Nov 03 14:30:08.321615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321594    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache" pod "bdd9d7fb-c03c-4814-9aee-6f3927d3e559" (UID: "bdd9d7fb-c03c-4814-9aee-6f3927d3e559"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:30:08.321883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321864    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:08.321947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321886    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:08.321947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.321901    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:08.323426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.323400    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs" pod "bdd9d7fb-c03c-4814-9aee-6f3927d3e559" (UID: "bdd9d7fb-c03c-4814-9aee-6f3927d3e559"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:30:08.323551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.323488    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp" pod "bdd9d7fb-c03c-4814-9aee-6f3927d3e559" (UID: "bdd9d7fb-c03c-4814-9aee-6f3927d3e559"). InnerVolumeSpecName "kube-api-access-gvcfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:08.323551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.323518    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm" pod "bdd9d7fb-c03c-4814-9aee-6f3927d3e559" (UID: "bdd9d7fb-c03c-4814-9aee-6f3927d3e559"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:30:08.423597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.423526    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:08.423597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.423552    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:08.423597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.423563    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvcfp\" (UniqueName: \"kubernetes.io/projected/bdd9d7fb-c03c-4814-9aee-6f3927d3e559-kube-api-access-gvcfp\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:08.574067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.574037    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:08.723008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.722927    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"
Nov 03 14:30:08.723008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.722949    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl" event={"ID":"bdd9d7fb-c03c-4814-9aee-6f3927d3e559","Type":"ContainerDied","Data":"f3906108ba605e6b85f24a426d8ca99e1bfe2d7a51d43284d2395dd156bf9f56"}
Nov 03 14:30:08.723008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.723001    6481 scope.go:122] "RemoveContainer" containerID="c33651932a7bacaf225c6ede35b7d47787b8485a4fc99118db6c251048b5be91"
Nov 03 14:30:08.777629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.777607    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"]
Nov 03 14:30:08.788952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:08.788923    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-2d0da2/rate-test-model-2d0da2-kserve-7fd6d67bfc-5rgvl"]
Nov 03 14:30:10.068032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:10.068001    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" path="/var/lib/kubelet/pods/bdd9d7fb-c03c-4814-9aee-6f3927d3e559/volumes"
Nov 03 14:30:10.579187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:10.579128    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:10.820570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:10.820512    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:11.522574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:11.522546    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:30:11.603732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:11.603698    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"]
Nov 03 14:30:12.588673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:12.588638    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:12.802627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:12.802590    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:13.011578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:13.011493    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:14.041741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:14.041711    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"]
Nov 03 14:30:14.052908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:14.052880    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16a15a83-ftb67"]
Nov 03 14:30:14.068137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:14.068112    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623ce384-d214-420d-aa2e-25d3b4955e24" path="/var/lib/kubelet/pods/623ce384-d214-420d-aa2e-25d3b4955e24/volumes"
Nov 03 14:30:14.634285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:14.634210    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:15.317127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317093    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"]
Nov 03 14:30:15.317721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317707    6481 cpu_manager.go:414] "containerMap: removing container" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main"
Nov 03 14:30:15.317766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317723    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main"
Nov 03 14:30:15.317766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317742    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5cd5f80-9d5a-4779-9f71-c874dbf99c51" containerName="revoke-keys"
Nov 03 14:30:15.317766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317747    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5cd5f80-9d5a-4779-9f71-c874dbf99c51" containerName="revoke-keys"
Nov 03 14:30:15.317882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317872    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5cd5f80-9d5a-4779-9f71-c874dbf99c51" containerName="revoke-keys"
Nov 03 14:30:15.317921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.317893    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd9d7fb-c03c-4814-9aee-6f3927d3e559" containerName="main"
Nov 03 14:30:15.322967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.322951    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.341632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.341606    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"]
Nov 03 14:30:15.348692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.348674    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:30:15.412999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.412968    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.413129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.413026    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.413174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.413135    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6wq\" (UniqueName: \"kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.514685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.514658    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.514814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.514702    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.514814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.514743    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6wq\" (UniqueName: \"kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.515473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.515452    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.516912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.516891    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.536120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.536097    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6wq\" (UniqueName: \"kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq\") pod \"maas-api-revoke-sub-a5de7cbe-zp6jb\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.636122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.636051    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:15.941374 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:15.941346    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22654ee6_b16f_4add_a55c_cbabe1f6674b.slice/crio-d4d37648f58e771a0764e2ab06853f39b6a4eae31d37b5cd22ba71e18098077c WatchSource:0}: Error finding container d4d37648f58e771a0764e2ab06853f39b6a4eae31d37b5cd22ba71e18098077c: Status 404 returned error can't find the container with id d4d37648f58e771a0764e2ab06853f39b6a4eae31d37b5cd22ba71e18098077c
Nov 03 14:30:15.942277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:15.942254    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"]
Nov 03 14:30:16.060770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.060731    6481 scope.go:122] "RemoveContainer" containerID="6b14976fcd1351b8d6584d1eab2f8706664bebcad6f9acd00d03e82283712c0d"
Nov 03 14:30:16.060936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.060867    6481 scope.go:122] "RemoveContainer" containerID="46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2"
Nov 03 14:30:16.060936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.060923    6481 scope.go:122] "RemoveContainer" containerID="9c061895dfaea69225da7df343c488cc42b7a4d74fb793f791d36417e34a297a"
Nov 03 14:30:16.061081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.061002    6481 scope.go:122] "RemoveContainer" containerID="3c4ad5a25d814a9ebd581ebadfc8057575b63e50ab7d710ee7dac1a22ec3630b"
Nov 03 14:30:16.129539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.129518    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:16.335155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.335126    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:16.604706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.604309    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt" podUID="770bc1b5-ca92-42a8-874f-da444a26c0f4" containerName="payload-processing" containerID="cri-o://e4a5f06440cefc71e779e0e7a1886b7e57ce403bd1eb2c0f5f82e90deebf20a6" gracePeriod=25
Nov 03 14:30:16.648476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.648452    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:16.796255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.794489    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerStarted","Data":"deb045b68ba1b769641c4bde0c09fa7866db260c7c3c6d91a3ffadeec66693f1"}
Nov 03 14:30:16.796438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.796406    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb" event={"ID":"22654ee6-b16f-4add-a55c-cbabe1f6674b","Type":"ContainerStarted","Data":"3f9e6e00fea7c270dbe3dd412647ede3311b85f99f33e181bbc6f15c178e4e36"}
Nov 03 14:30:16.796581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.796466    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb" event={"ID":"22654ee6-b16f-4add-a55c-cbabe1f6674b","Type":"ContainerStarted","Data":"d4d37648f58e771a0764e2ab06853f39b6a4eae31d37b5cd22ba71e18098077c"}
Nov 03 14:30:16.799023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.798983    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerStarted","Data":"bf7ea463d85163f8567e89607b25070aef76abdabc8a33d4fa3507e52ee14b5c"}
Nov 03 14:30:16.805021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.802383    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerStarted","Data":"73fff16eed388fd458573f89a1c4b518c67fa8e217ce9a36f20e63c0eaa523ac"}
Nov 03 14:30:16.807022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.807000    6481 generic.go:356] "Generic (PLEG): container finished" podID="770bc1b5-ca92-42a8-874f-da444a26c0f4" containerID="e4a5f06440cefc71e779e0e7a1886b7e57ce403bd1eb2c0f5f82e90deebf20a6" exitCode=0
Nov 03 14:30:16.807136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.807083    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt" event={"ID":"770bc1b5-ca92-42a8-874f-da444a26c0f4","Type":"ContainerDied","Data":"e4a5f06440cefc71e779e0e7a1886b7e57ce403bd1eb2c0f5f82e90deebf20a6"}
Nov 03 14:30:16.814652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.813180    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerStarted","Data":"c913e25689e0dd968ae0d174aa09da1d4cc5e11dbd5cd250e725c63003cec1f7"}
Nov 03 14:30:16.873186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.873161    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:30:16.933844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.933809    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume\") pod \"770bc1b5-ca92-42a8-874f-da444a26c0f4\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") "
Nov 03 14:30:16.933989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.933892    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume\") pod \"770bc1b5-ca92-42a8-874f-da444a26c0f4\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") "
Nov 03 14:30:16.933989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.933957    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials\") pod \"770bc1b5-ca92-42a8-874f-da444a26c0f4\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") "
Nov 03 14:30:16.934112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.934047    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt\") pod \"770bc1b5-ca92-42a8-874f-da444a26c0f4\" (UID: \"770bc1b5-ca92-42a8-874f-da444a26c0f4\") "
Nov 03 14:30:16.934349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.934292    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume" pod "770bc1b5-ca92-42a8-874f-da444a26c0f4" (UID: "770bc1b5-ca92-42a8-874f-da444a26c0f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:16.934349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.934296    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume" pod "770bc1b5-ca92-42a8-874f-da444a26c0f4" (UID: "770bc1b5-ca92-42a8-874f-da444a26c0f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:16.934760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.934717    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:16.934760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.934742    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/770bc1b5-ca92-42a8-874f-da444a26c0f4-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:16.936011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.935992    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials" pod "770bc1b5-ca92-42a8-874f-da444a26c0f4" (UID: "770bc1b5-ca92-42a8-874f-da444a26c0f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:16.936131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.936117    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt" pod "770bc1b5-ca92-42a8-874f-da444a26c0f4" (UID: "770bc1b5-ca92-42a8-874f-da444a26c0f4"). InnerVolumeSpecName "kube-api-access-rw7tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:16.967633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:16.967585    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb" podStartSLOduration=1.967571048 podStartE2EDuration="1.967571048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:16.966323869 +0000 UTC m=+1095.016355056" watchObservedRunningTime="2026-11-03 14:30:16.967571048 +0000 UTC m=+1095.017602231"
Nov 03 14:30:17.035900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.035875    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:17.035996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.035904    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw7tt\" (UniqueName: \"kubernetes.io/projected/770bc1b5-ca92-42a8-874f-da444a26c0f4-kube-api-access-rw7tt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:17.050533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.050505    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"]
Nov 03 14:30:17.062351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.062327    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab3b030e-jjrcd"]
Nov 03 14:30:17.511413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.511375    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"]
Nov 03 14:30:17.512036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.512019    6481 cpu_manager.go:414] "containerMap: removing container" podUID="770bc1b5-ca92-42a8-874f-da444a26c0f4" containerName="payload-processing"
Nov 03 14:30:17.512096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.512036    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770bc1b5-ca92-42a8-874f-da444a26c0f4" containerName="payload-processing"
Nov 03 14:30:17.512153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.512143    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="770bc1b5-ca92-42a8-874f-da444a26c0f4" containerName="payload-processing"
Nov 03 14:30:17.515513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.515488    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.533599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.533570    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"]
Nov 03 14:30:17.538470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.538446    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.541417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541392    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.541533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541503    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.541638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541619    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.541715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541698    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.541787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541770    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.541950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541933    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.542028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.541979    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.542142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.542118    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.542215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.542170    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87lm\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.542584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.542472    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"]
Nov 03 14:30:17.571431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.567423    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"]
Nov 03 14:30:17.643170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643135    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643178    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.643356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643222    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643325    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.643522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643373    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643448    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87lm\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643495    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.643643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643526    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.643643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643603    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643648    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643672    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643762    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643776    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643805    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643878    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9z57\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.643966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643905    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.643966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643914    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.644081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.643923    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.644081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.644031    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.644081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.644053    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.644247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.644101    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.644247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.644157    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.644358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.644269    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.646032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.646014    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.646460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.646441    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.664576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.664552    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.664691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.664668    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87lm\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.745756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.745722    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.745926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.745789    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.745926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.745836    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.745926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.745875    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.745926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.745924    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.745952    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746100    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9z57\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746242    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746331    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746454 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746132    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746503    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746553    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746632    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.746837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.746817    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.748049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.748030    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.748370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.748347    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.766374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.766317    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9z57\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.766547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.766527    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.822590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.822562    6481 generic.go:356] "Generic (PLEG): container finished" podID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerID="deb045b68ba1b769641c4bde0c09fa7866db260c7c3c6d91a3ffadeec66693f1" exitCode=60
Nov 03 14:30:17.822722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.822639    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerDied","Data":"deb045b68ba1b769641c4bde0c09fa7866db260c7c3c6d91a3ffadeec66693f1"}
Nov 03 14:30:17.822799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.822722    6481 scope.go:122] "RemoveContainer" containerID="9c061895dfaea69225da7df343c488cc42b7a4d74fb793f791d36417e34a297a"
Nov 03 14:30:17.823134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.823111    6481 scope.go:122] "RemoveContainer" containerID="deb045b68ba1b769641c4bde0c09fa7866db260c7c3c6d91a3ffadeec66693f1"
Nov 03 14:30:17.823492 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:17.823473    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l_odh-ai-gateway-infra(27a364cb-17b8-4e41-a2aa-0eddf12f741a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a"
Nov 03 14:30:17.824471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.824444    6481 generic.go:356] "Generic (PLEG): container finished" podID="22654ee6-b16f-4add-a55c-cbabe1f6674b" containerID="3f9e6e00fea7c270dbe3dd412647ede3311b85f99f33e181bbc6f15c178e4e36" exitCode=0
Nov 03 14:30:17.824567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.824505    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb" event={"ID":"22654ee6-b16f-4add-a55c-cbabe1f6674b","Type":"ContainerDied","Data":"3f9e6e00fea7c270dbe3dd412647ede3311b85f99f33e181bbc6f15c178e4e36"}
Nov 03 14:30:17.826288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.826267    6481 generic.go:356] "Generic (PLEG): container finished" podID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerID="bf7ea463d85163f8567e89607b25070aef76abdabc8a33d4fa3507e52ee14b5c" exitCode=60
Nov 03 14:30:17.826387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.826336    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerDied","Data":"bf7ea463d85163f8567e89607b25070aef76abdabc8a33d4fa3507e52ee14b5c"}
Nov 03 14:30:17.826674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.826658    6481 scope.go:122] "RemoveContainer" containerID="bf7ea463d85163f8567e89607b25070aef76abdabc8a33d4fa3507e52ee14b5c"
Nov 03 14:30:17.826948 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:17.826928    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh_odh-ai-gateway-infra(b00777ec-effb-4bbe-a9e0-0a3dea4c2c78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78"
Nov 03 14:30:17.828470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.828449    6481 generic.go:356] "Generic (PLEG): container finished" podID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerID="73fff16eed388fd458573f89a1c4b518c67fa8e217ce9a36f20e63c0eaa523ac" exitCode=60
Nov 03 14:30:17.828553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.828521    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerDied","Data":"73fff16eed388fd458573f89a1c4b518c67fa8e217ce9a36f20e63c0eaa523ac"}
Nov 03 14:30:17.828837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.828823    6481 scope.go:122] "RemoveContainer" containerID="73fff16eed388fd458573f89a1c4b518c67fa8e217ce9a36f20e63c0eaa523ac"
Nov 03 14:30:17.829107 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:17.829086    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9_odh-ai-gateway-infra(2690fc59-61ca-4197-894b-8a4c2754f1f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6"
Nov 03 14:30:17.829978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.829956    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt" event={"ID":"770bc1b5-ca92-42a8-874f-da444a26c0f4","Type":"ContainerDied","Data":"aa7964494ed01b455c41041b53b4b10870a6ee22155c57d12910a8afe4195ce6"}
Nov 03 14:30:17.830081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.829962    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"
Nov 03 14:30:17.831956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.831931    6481 generic.go:356] "Generic (PLEG): container finished" podID="167425fa-3244-4a80-8630-80b0d687ce4b" containerID="c913e25689e0dd968ae0d174aa09da1d4cc5e11dbd5cd250e725c63003cec1f7" exitCode=60
Nov 03 14:30:17.832074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.831994    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerDied","Data":"c913e25689e0dd968ae0d174aa09da1d4cc5e11dbd5cd250e725c63003cec1f7"}
Nov 03 14:30:17.832330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.832310    6481 scope.go:122] "RemoveContainer" containerID="c913e25689e0dd968ae0d174aa09da1d4cc5e11dbd5cd250e725c63003cec1f7"
Nov 03 14:30:17.832573 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:17.832554    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd_odh-ai-gateway-infra(167425fa-3244-4a80-8630-80b0d687ce4b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" podUID="167425fa-3244-4a80-8630-80b0d687ce4b"
Nov 03 14:30:17.838350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.838278    6481 scope.go:122] "RemoveContainer" containerID="46538cf206d3af6aff7f723b2e892960cde73944797d5a5585390dc51b60a0c2"
Nov 03 14:30:17.852651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.852629    6481 scope.go:122] "RemoveContainer" containerID="6b14976fcd1351b8d6584d1eab2f8706664bebcad6f9acd00d03e82283712c0d"
Nov 03 14:30:17.855565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.855546    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"]
Nov 03 14:30:17.865880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.865865    6481 scope.go:122] "RemoveContainer" containerID="e4a5f06440cefc71e779e0e7a1886b7e57ce403bd1eb2c0f5f82e90deebf20a6"
Nov 03 14:30:17.871930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.871913    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:17.875081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.875057    6481 scope.go:122] "RemoveContainer" containerID="3c4ad5a25d814a9ebd581ebadfc8057575b63e50ab7d710ee7dac1a22ec3630b"
Nov 03 14:30:17.875184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.875158    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:17.898874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.898852    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"]
Nov 03 14:30:17.950278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:17.950203    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"]
Nov 03 14:30:18.033809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.033736    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"]
Nov 03 14:30:18.070420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.070393    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bf7d5e-cde5-4ee0-8d27-5597ba64fc09" path="/var/lib/kubelet/pods/09bf7d5e-cde5-4ee0-8d27-5597ba64fc09/volumes"
Nov 03 14:30:18.124061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.124031    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"]
Nov 03 14:30:18.132441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.132411    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-59b8bbprt"]
Nov 03 14:30:18.205567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.205538    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"]
Nov 03 14:30:18.206269 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:18.206206    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51173fe0_38c4_4a2e_948c_f595deedce2b.slice/crio-9159bd7ce9885385a2f62a17cf3883ebb7ae353458a740830484ab17dc10359e WatchSource:0}: Error finding container 9159bd7ce9885385a2f62a17cf3883ebb7ae353458a740830484ab17dc10359e: Status 404 returned error can't find the container with id 9159bd7ce9885385a2f62a17cf3883ebb7ae353458a740830484ab17dc10359e
Nov 03 14:30:18.209093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.209064    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:18.209176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.209127    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:18.209176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.209170    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:18.209649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.209610    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"]
Nov 03 14:30:18.223282 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:18.223258    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0b0ce3_9e5f_4670_9af0_53a27c4b9ec3.slice/crio-39aead63c7adbf854ac5baa669ce5e406c340eb3a93a90aba5bc0f3b9779a2bd WatchSource:0}: Error finding container 39aead63c7adbf854ac5baa669ce5e406c340eb3a93a90aba5bc0f3b9779a2bd: Status 404 returned error can't find the container with id 39aead63c7adbf854ac5baa669ce5e406c340eb3a93a90aba5bc0f3b9779a2bd
Nov 03 14:30:18.225774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.225743    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:18.225862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.225804    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:18.225862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.225845    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:18.798446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.798405    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:18.843886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.843847    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" event={"ID":"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3","Type":"ContainerStarted","Data":"f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266"}
Nov 03 14:30:18.844012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.843906    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" event={"ID":"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3","Type":"ContainerStarted","Data":"39aead63c7adbf854ac5baa669ce5e406c340eb3a93a90aba5bc0f3b9779a2bd"}
Nov 03 14:30:18.853003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.852966    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" event={"ID":"51173fe0-38c4-4a2e-948c-f595deedce2b","Type":"ContainerStarted","Data":"258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03"}
Nov 03 14:30:18.853137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.853031    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" event={"ID":"51173fe0-38c4-4a2e-948c-f595deedce2b","Type":"ContainerStarted","Data":"9159bd7ce9885385a2f62a17cf3883ebb7ae353458a740830484ab17dc10359e"}
Nov 03 14:30:18.872517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.872414    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:18.875997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.875923    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:18.934554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:18.934493    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podStartSLOduration=1.9344703829999998 podStartE2EDuration="1.934470383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:18.929988455 +0000 UTC m=+1096.980019641" watchObservedRunningTime="2026-11-03 14:30:18.934470383 +0000 UTC m=+1096.984501566"
Nov 03 14:30:19.112475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.112411    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podStartSLOduration=2.112386921 podStartE2EDuration="2.112386921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:19.06812472 +0000 UTC m=+1097.118155904" watchObservedRunningTime="2026-11-03 14:30:19.112386921 +0000 UTC m=+1097.162418106"
Nov 03 14:30:19.163009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.162262    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:19.179909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.179879    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn\" (UniqueName: \"kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn\") pod \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") "
Nov 03 14:30:19.180064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.180008    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca\") pod \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\" (UID: \"27a364cb-17b8-4e41-a2aa-0eddf12f741a\") "
Nov 03 14:30:19.182638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.181408    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca" pod "27a364cb-17b8-4e41-a2aa-0eddf12f741a" (UID: "27a364cb-17b8-4e41-a2aa-0eddf12f741a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:19.196276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.192143    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn" pod "27a364cb-17b8-4e41-a2aa-0eddf12f741a" (UID: "27a364cb-17b8-4e41-a2aa-0eddf12f741a"). InnerVolumeSpecName "kube-api-access-b4scn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:19.282764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.282723    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4scn\" (UniqueName: \"kubernetes.io/projected/27a364cb-17b8-4e41-a2aa-0eddf12f741a-kube-api-access-b4scn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.282764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.282762    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a364cb-17b8-4e41-a2aa-0eddf12f741a-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.285899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.285873    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:19.288182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.288153    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:19.561119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.537712    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq\" (UniqueName: \"kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq\") pod \"22654ee6-b16f-4add-a55c-cbabe1f6674b\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") "
Nov 03 14:30:19.561119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.537871    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca\") pod \"167425fa-3244-4a80-8630-80b0d687ce4b\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") "
Nov 03 14:30:19.561119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.538105    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl\" (UniqueName: \"kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl\") pod \"167425fa-3244-4a80-8630-80b0d687ce4b\" (UID: \"167425fa-3244-4a80-8630-80b0d687ce4b\") "
Nov 03 14:30:19.561119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.538367    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca\") pod \"22654ee6-b16f-4add-a55c-cbabe1f6674b\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") "
Nov 03 14:30:19.561119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.538620    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca\") pod \"22654ee6-b16f-4add-a55c-cbabe1f6674b\" (UID: \"22654ee6-b16f-4add-a55c-cbabe1f6674b\") "
Nov 03 14:30:19.565765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.565731    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca" pod "22654ee6-b16f-4add-a55c-cbabe1f6674b" (UID: "22654ee6-b16f-4add-a55c-cbabe1f6674b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:19.572442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.572409    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca" pod "167425fa-3244-4a80-8630-80b0d687ce4b" (UID: "167425fa-3244-4a80-8630-80b0d687ce4b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:19.576592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.576566    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl" pod "167425fa-3244-4a80-8630-80b0d687ce4b" (UID: "167425fa-3244-4a80-8630-80b0d687ce4b"). InnerVolumeSpecName "kube-api-access-xtvjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:19.584491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.584451    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq" pod "22654ee6-b16f-4add-a55c-cbabe1f6674b" (UID: "22654ee6-b16f-4add-a55c-cbabe1f6674b"). InnerVolumeSpecName "kube-api-access-qm6wq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:19.584737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.584718    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca" pod "22654ee6-b16f-4add-a55c-cbabe1f6674b" (UID: "22654ee6-b16f-4add-a55c-cbabe1f6674b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:30:19.658325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.650451    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/167425fa-3244-4a80-8630-80b0d687ce4b-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.659453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.659415    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtvjl\" (UniqueName: \"kubernetes.io/projected/167425fa-3244-4a80-8630-80b0d687ce4b-kube-api-access-xtvjl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.659660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.659643    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.659756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.659744    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22654ee6-b16f-4add-a55c-cbabe1f6674b-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.659841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.659830    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6wq\" (UniqueName: \"kubernetes.io/projected/22654ee6-b16f-4add-a55c-cbabe1f6674b-kube-api-access-qm6wq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.666291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.665955    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:19.668714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.668687    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:19.866141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.866096    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca\") pod \"2690fc59-61ca-4197-894b-8a4c2754f1f6\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") "
Nov 03 14:30:19.866849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.866828    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4\" (UniqueName: \"kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4\") pod \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") "
Nov 03 14:30:19.867015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.867000    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc\" (UniqueName: \"kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc\") pod \"2690fc59-61ca-4197-894b-8a4c2754f1f6\" (UID: \"2690fc59-61ca-4197-894b-8a4c2754f1f6\") "
Nov 03 14:30:19.867208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.867190    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca\") pod \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\" (UID: \"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78\") "
Nov 03 14:30:19.868150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.868128    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca" pod "b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" (UID: "b00777ec-effb-4bbe-a9e0-0a3dea4c2c78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:19.868624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.868607    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca" pod "2690fc59-61ca-4197-894b-8a4c2754f1f6" (UID: "2690fc59-61ca-4197-894b-8a4c2754f1f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:19.873281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.872159    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc" pod "2690fc59-61ca-4197-894b-8a4c2754f1f6" (UID: "2690fc59-61ca-4197-894b-8a4c2754f1f6"). InnerVolumeSpecName "kube-api-access-h5tsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:19.875253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.873688    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:19.875253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.873742    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:19.877121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.877024    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:19.877121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.877070    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:19.877294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.877145    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4" pod "b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" (UID: "b00777ec-effb-4bbe-a9e0-0a3dea4c2c78"). InnerVolumeSpecName "kube-api-access-d6sg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:19.880930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.880908    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9" event={"ID":"2690fc59-61ca-4197-894b-8a4c2754f1f6","Type":"ContainerDied","Data":"0ea1e99312ac67cb46c5223ff8b6a768e4d7764126ea601e7b7f536297b25ed0"}
Nov 03 14:30:19.881020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.880999    6481 scope.go:122] "RemoveContainer" containerID="73fff16eed388fd458573f89a1c4b518c67fa8e217ce9a36f20e63c0eaa523ac"
Nov 03 14:30:19.881127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.881112    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"
Nov 03 14:30:19.883768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.883734    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd" event={"ID":"167425fa-3244-4a80-8630-80b0d687ce4b","Type":"ContainerDied","Data":"a55bfc5896d35ed78149bfb7611456d497fe6b0dc2058aa42afb54008ed3d16d"}
Nov 03 14:30:19.884356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.883857    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"
Nov 03 14:30:19.887211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.887180    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l" event={"ID":"27a364cb-17b8-4e41-a2aa-0eddf12f741a","Type":"ContainerDied","Data":"99b5dc240ed0993f8a14edee2a7046b99b4319d1e46ce0bb4ba36f8c280cf67a"}
Nov 03 14:30:19.887330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.887308    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"
Nov 03 14:30:19.889731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.889707    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb" event={"ID":"22654ee6-b16f-4add-a55c-cbabe1f6674b","Type":"ContainerDied","Data":"d4d37648f58e771a0764e2ab06853f39b6a4eae31d37b5cd22ba71e18098077c"}
Nov 03 14:30:19.889843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.889744    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d37648f58e771a0764e2ab06853f39b6a4eae31d37b5cd22ba71e18098077c"
Nov 03 14:30:19.889843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.889802    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"
Nov 03 14:30:19.893491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.893190    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"
Nov 03 14:30:19.893491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.893187    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh" event={"ID":"b00777ec-effb-4bbe-a9e0-0a3dea4c2c78","Type":"ContainerDied","Data":"0f301bda14448051cf1028abb6e1316b9b54ddeea41414fc314c9b5689c22266"}
Nov 03 14:30:19.965524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.965472    6481 scope.go:122] "RemoveContainer" containerID="c913e25689e0dd968ae0d174aa09da1d4cc5e11dbd5cd250e725c63003cec1f7"
Nov 03 14:30:19.973674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.971461    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2690fc59-61ca-4197-894b-8a4c2754f1f6-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.973674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.971489    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6sg4\" (UniqueName: \"kubernetes.io/projected/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-kube-api-access-d6sg4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.973674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.971505    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5tsc\" (UniqueName: \"kubernetes.io/projected/2690fc59-61ca-4197-894b-8a4c2754f1f6-kube-api-access-h5tsc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.973674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.971519    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:19.987507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.987485    6481 scope.go:122] "RemoveContainer" containerID="deb045b68ba1b769641c4bde0c09fa7866db260c7c3c6d91a3ffadeec66693f1"
Nov 03 14:30:19.996732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:19.996709    6481 scope.go:122] "RemoveContainer" containerID="bf7ea463d85163f8567e89607b25070aef76abdabc8a33d4fa3507e52ee14b5c"
Nov 03 14:30:20.073147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.073115    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770bc1b5-ca92-42a8-874f-da444a26c0f4" path="/var/lib/kubelet/pods/770bc1b5-ca92-42a8-874f-da444a26c0f4/volumes"
Nov 03 14:30:20.082248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.080456    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"]
Nov 03 14:30:20.087256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.087198    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-86c2e2-29895270-hdct9"]
Nov 03 14:30:20.434551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.432579    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"]
Nov 03 14:30:20.480253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.479292    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895270-5r5xh"]
Nov 03 14:30:20.515372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.515280    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"]
Nov 03 14:30:20.567364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.567286    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2d0da2-29895270-2jt9l"]
Nov 03 14:30:20.572300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.572272    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"]
Nov 03 14:30:20.607253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.583330    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-156ae1-29895270-8bgmd"]
Nov 03 14:30:20.877309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.875823    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:20.877309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.875883    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:20.877309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.876636    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:20.877309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:20.876718    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:21.180573 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.180538    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:21.231152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.230575    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Nov 03 14:30:21.435310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.434314    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:21.883623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.874587    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:21.883623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.874653    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:21.884991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.884535    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:21.884991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:21.884593    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:22.127506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.127457    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Nov 03 14:30:22.213261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.180473    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:30:22.221254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.213694    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:22.249252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.248584    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:22.447371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.447313    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:22.585317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.579573    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" path="/var/lib/kubelet/pods/167425fa-3244-4a80-8630-80b0d687ce4b/volumes"
Nov 03 14:30:22.609015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.608976    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" path="/var/lib/kubelet/pods/2690fc59-61ca-4197-894b-8a4c2754f1f6/volumes"
Nov 03 14:30:22.617283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.610125    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" path="/var/lib/kubelet/pods/27a364cb-17b8-4e41-a2aa-0eddf12f741a/volumes"
Nov 03 14:30:22.626423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.626383    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" path="/var/lib/kubelet/pods/b00777ec-effb-4bbe-a9e0-0a3dea4c2c78/volumes"
Nov 03 14:30:22.632604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.632583    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:30:22.632850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.632835    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:30:22.899311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.898367    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:22.899311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.898439    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:22.921265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.913783    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:22.921265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:22.913879    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:23.128355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.128313    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:23.129030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.129006    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:30:23.201254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.186288    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:23.201871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.201838    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:30:23.228774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.228737    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:30:23.229558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.229536    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:30:23.290282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.256021    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:23.437268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.437202    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:23.462426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.462321    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:23.709251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.705757    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:23.874630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.874540    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:23.887738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.887695    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:23.903554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.903521    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:23.904063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.904036    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:23.938946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:23.938913    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:24.143304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.133484    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:24.172173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.172139    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:24.204402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.204353    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:24.233526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.233410    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:24.255130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.254061    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:30:24.339375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.339332    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:24.341917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.341063    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Nov 03 14:30:24.483480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.483433    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded"
Nov 03 14:30:24.493337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.493311    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts"
Nov 03 14:30:24.581427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.581385    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:30:24.589086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.589063    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:30:24.601248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.592849    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:30:24.872507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.872422    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:24.872775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.872741    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:24.875899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.875872    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:24.876024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:24.875916    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:25.584253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:25.581035    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:25.873288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:25.873181    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Nov 03 14:30:25.873288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:25.873266    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Nov 03 14:30:25.875494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:25.875467    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body=
Nov 03 14:30:25.875613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:25.875511    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused"
Nov 03 14:30:26.904485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.904443    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"]
Nov 03 14:30:26.905355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905334    6481 cpu_manager.go:414] "containerMap: removing container" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.905355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905353    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905367    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905376    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905392    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905399    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905412    6481 cpu_manager.go:414] "containerMap: removing container" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905419    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905448    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905455    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905495    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905502    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905514    6481 cpu_manager.go:414] "containerMap: removing container" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905522    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905533    6481 cpu_manager.go:414] "containerMap: removing container" podUID="22654ee6-b16f-4add-a55c-cbabe1f6674b" containerName="revoke-keys"
Nov 03 14:30:26.905546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905541    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22654ee6-b16f-4add-a55c-cbabe1f6674b" containerName="revoke-keys"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905560    6481 cpu_manager.go:414] "containerMap: removing container" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905567    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905689    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905701    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905713    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905722    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="22654ee6-b16f-4add-a55c-cbabe1f6674b" containerName="revoke-keys"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905735    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905745    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905753    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905762    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905773    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905783    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:26.906323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.905793    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:26.911771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.911744    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:26.936671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.936648    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:30:26.936922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.936905    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:30:26.937104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.937088    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:30:26.946321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.946291    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"]
Nov 03 14:30:26.947525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:26.947488    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:30:27.072528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.072492    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj68\" (UniqueName: \"kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.072709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.072544    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.072709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.072567    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.173720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.173684    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.173953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.173934    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj68\" (UniqueName: \"kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.174051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.174022    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.174683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.174660    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.176261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.176207    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.196078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.196040    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj68\" (UniqueName: \"kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68\") pod \"maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.245283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.245252    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:27.571641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.571604    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"]
Nov 03 14:30:27.630247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.629140    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:27.631019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.630642    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:27.637291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.633622    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:30:27.692561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.692531    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6" event={"ID":"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf","Type":"ContainerStarted","Data":"f21085befa90569c2e1d5452e9261b58b09d02c8351fa6742ec8f9acff19876e"}
Nov 03 14:30:27.754271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.752705    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:27.754271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.753385    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:27.754271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.754202    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:30:27.814810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:27.814785    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:27.989509 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:27.989471    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:30:28.701077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:28.701044    6481 generic.go:356] "Generic (PLEG): container finished" podID="d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" containerID="39508baf1b82f7b06bf2e6e108689b290422bbd33c833b51f18b4802cdce6ee9" exitCode=0
Nov 03 14:30:28.701286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:28.701204    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6" event={"ID":"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf","Type":"ContainerDied","Data":"39508baf1b82f7b06bf2e6e108689b290422bbd33c833b51f18b4802cdce6ee9"}
Nov 03 14:30:29.280174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.280142    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:29.500033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.499993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:29.864241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.864210    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:29.915633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.915610    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68\" (UniqueName: \"kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68\") pod \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") "
Nov 03 14:30:29.915804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.915665    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca\") pod \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") "
Nov 03 14:30:29.915804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.915746    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca\") pod \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\" (UID: \"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf\") "
Nov 03 14:30:29.916677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.916629    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca" pod "d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" (UID: "d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:29.916965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.916936    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:29.918431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.918410    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68" pod "d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" (UID: "d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf"). InnerVolumeSpecName "kube-api-access-6qj68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:29.918689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.918662    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca" pod "d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" (UID: "d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:30:29.996311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:29.996163    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:30.018217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.018178    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qj68\" (UniqueName: \"kubernetes.io/projected/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-kube-api-access-6qj68\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:30.018217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.018208    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:30.632492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.632460    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"]
Nov 03 14:30:30.633056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633041    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" containerName="revoke-keys"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633056    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" containerName="revoke-keys"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633071    6481 cpu_manager.go:414] "containerMap: removing container" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633076    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="167425fa-3244-4a80-8630-80b0d687ce4b" containerName="cleanup"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633088    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633093    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2690fc59-61ca-4197-894b-8a4c2754f1f6" containerName="cleanup"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633101    6481 cpu_manager.go:414] "containerMap: removing container" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:30.633110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633105    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:30.633380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633121    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:30.633380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633126    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:30.633380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633217    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b00777ec-effb-4bbe-a9e0-0a3dea4c2c78" containerName="cleanup"
Nov 03 14:30:30.633380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633238    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a364cb-17b8-4e41-a2aa-0eddf12f741a" containerName="cleanup"
Nov 03 14:30:30.633380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.633250    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" containerName="revoke-keys"
Nov 03 14:30:30.636966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.636951    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.668377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.668348    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"]
Nov 03 14:30:30.717120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.717097    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"
Nov 03 14:30:30.717271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.717142    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6" event={"ID":"d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf","Type":"ContainerDied","Data":"f21085befa90569c2e1d5452e9261b58b09d02c8351fa6742ec8f9acff19876e"}
Nov 03 14:30:30.717271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.717190    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f21085befa90569c2e1d5452e9261b58b09d02c8351fa6742ec8f9acff19876e"
Nov 03 14:30:30.728196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.728173    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.728352 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.728247    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.728434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.728411    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7khf\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.728536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.728515    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.829166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.829138    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7khf\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.829343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.829185    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.829343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.829283    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.829343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.829337    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.829958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.829937    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.830104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.830084    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.831788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.831769    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.853804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.853780    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7khf\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:30.950414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:30.950377    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:31.256074 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:31.256043    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67feb46_a095_4ea5_97e6_8d60f999bce1.slice/crio-d772f8c25039c2495c3051d0edbfe4189824d25091eb35bf70b5959918aed1ca WatchSource:0}: Error finding container d772f8c25039c2495c3051d0edbfe4189824d25091eb35bf70b5959918aed1ca: Status 404 returned error can't find the container with id d772f8c25039c2495c3051d0edbfe4189824d25091eb35bf70b5959918aed1ca
Nov 03 14:30:31.256389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:31.256366    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"]
Nov 03 14:30:31.733967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:31.733884    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m" event={"ID":"b67feb46-a095-4ea5-97e6-8d60f999bce1","Type":"ContainerStarted","Data":"641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d"}
Nov 03 14:30:31.734434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:31.734030    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:31.734434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:31.734059    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m" event={"ID":"b67feb46-a095-4ea5-97e6-8d60f999bce1","Type":"ContainerStarted","Data":"d772f8c25039c2495c3051d0edbfe4189824d25091eb35bf70b5959918aed1ca"}
Nov 03 14:30:31.780959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:31.780910    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m" podStartSLOduration=1.7808986409999998 podStartE2EDuration="1.780898641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:31.775416793 +0000 UTC m=+1109.825447976" watchObservedRunningTime="2026-11-03 14:30:31.780898641 +0000 UTC m=+1109.830929824"
Nov 03 14:30:32.109438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:32.109340    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:33.096048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:33.096009    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:33.578067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:33.578022    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:34.113471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:34.113437    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:35.043883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.043849    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"]
Nov 03 14:30:35.055645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.055616    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55746ebb-j6566"]
Nov 03 14:30:35.350214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.350142    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"]
Nov 03 14:30:35.356527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.356498    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.377074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.377047    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"]
Nov 03 14:30:35.382132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.382099    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.382280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.382167    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.382368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.382344    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlmj\" (UniqueName: \"kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.384028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.384002    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:30:35.384135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.384026    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:30:35.384135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.384087    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:30:35.384300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.384019    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:30:35.483305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.483274    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.483480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.483319    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.483480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.483371    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlmj\" (UniqueName: \"kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.484037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.484017    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.485637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.485615    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.506504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.506479    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlmj\" (UniqueName: \"kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj\") pod \"maas-api-revoke-sub-7733e9cb-fgxsp\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.676433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.676403    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:35.979375 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:35.979347    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb011c3_b2ba_408a_bd16_2cc8497b0ca2.slice/crio-65430d0c3f382e12963bc74a9a364042a4aea5dc73a87e796ccf003f7ce8e6f3 WatchSource:0}: Error finding container 65430d0c3f382e12963bc74a9a364042a4aea5dc73a87e796ccf003f7ce8e6f3: Status 404 returned error can't find the container with id 65430d0c3f382e12963bc74a9a364042a4aea5dc73a87e796ccf003f7ce8e6f3
Nov 03 14:30:35.980508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:35.980486    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"]
Nov 03 14:30:36.076996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:36.076962    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e93b4f-b94b-4683-bb07-d5063fe45c06" path="/var/lib/kubelet/pods/92e93b4f-b94b-4683-bb07-d5063fe45c06/volumes"
Nov 03 14:30:36.097070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:36.097043    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:36.775687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:36.775650    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp" event={"ID":"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2","Type":"ContainerStarted","Data":"c84df38834f0fb58af937ba7f20fbd421c4d43ff78a540db9602364a63b0e297"}
Nov 03 14:30:36.776365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:36.775710    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp" event={"ID":"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2","Type":"ContainerStarted","Data":"65430d0c3f382e12963bc74a9a364042a4aea5dc73a87e796ccf003f7ce8e6f3"}
Nov 03 14:30:36.816722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:36.816661    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp" podStartSLOduration=1.816646497 podStartE2EDuration="1.816646497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:36.815133853 +0000 UTC m=+1114.865165037" watchObservedRunningTime="2026-11-03 14:30:36.816646497 +0000 UTC m=+1114.866677681"
Nov 03 14:30:37.190900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.190868    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:30:37.315970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.315932    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"]
Nov 03 14:30:37.321332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.321306    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.343489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.343459    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"]
Nov 03 14:30:37.348915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.348888    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4a67e25f-openshift-default-dockercfg-v66ww\""
Nov 03 14:30:37.411548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411521    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411568    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411649    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8jsm\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411749    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411775    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411803    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411908    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411950    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.411990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.411980    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.512986    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513068    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513126    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513163    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513348    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513417    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513430    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513461    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513508    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8jsm\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513589    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513608    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513695    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.513983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.513958    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.514299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.514279    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.515651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.515633    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.515850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.515831    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.538023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.537944    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.539281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.539257    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8jsm\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm\") pod \"e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") " pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.636802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.636772    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:37.637688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.637669    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:37.791609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.791526    6481 generic.go:356] "Generic (PLEG): container finished" podID="bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" containerID="c84df38834f0fb58af937ba7f20fbd421c4d43ff78a540db9602364a63b0e297" exitCode=0
Nov 03 14:30:37.791983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.791628    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp" event={"ID":"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2","Type":"ContainerDied","Data":"c84df38834f0fb58af937ba7f20fbd421c4d43ff78a540db9602364a63b0e297"}
Nov 03 14:30:37.905253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:37.901080    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:38.015485 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:38.013554    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaae572f6_eb7d_41d3_8b2e_1b6262a5d5e2.slice/crio-283c45443493edda5002a67030e6561669328e4b542ab20fc0461e5c075c4f68 WatchSource:0}: Error finding container 283c45443493edda5002a67030e6561669328e4b542ab20fc0461e5c075c4f68: Status 404 returned error can't find the container with id 283c45443493edda5002a67030e6561669328e4b542ab20fc0461e5c075c4f68
Nov 03 14:30:38.017852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.017147    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"]
Nov 03 14:30:38.018514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.018378    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:38.018629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.018522    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:38.018629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.018573    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:30:38.047843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.047741    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"]
Nov 03 14:30:38.054219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.054192    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a92ede12-jwjzc"]
Nov 03 14:30:38.073317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.073295    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee5a510-faca-4805-b851-48030c4ff04f" path="/var/lib/kubelet/pods/8ee5a510-faca-4805-b851-48030c4ff04f/volumes"
Nov 03 14:30:38.247306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.247184    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:38.466252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.465555    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:38.725256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.723421    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:38.967254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.946603    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" event={"ID":"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2","Type":"ContainerStarted","Data":"ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1"}
Nov 03 14:30:38.968319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:38.968286    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" event={"ID":"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2","Type":"ContainerStarted","Data":"283c45443493edda5002a67030e6561669328e4b542ab20fc0461e5c075c4f68"}
Nov 03 14:30:39.231254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:39.208973    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podStartSLOduration=2.208948147 podStartE2EDuration="2.208948147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:39.026599796 +0000 UTC m=+1117.076630980" watchObservedRunningTime="2026-11-03 14:30:39.208948147 +0000 UTC m=+1117.258979330"
Nov 03 14:30:39.585337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:39.585132    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:39.640513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:39.638592    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:39.641123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:39.640799    6481 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 14:30:39.641123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:39.640855    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 14:30:40.247266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.238305    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": context deadline exceeded"
Nov 03 14:30:40.257390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.256523    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:40.317566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.317474    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:30:40.328507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.328464    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:40.498028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.493077    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:40.504769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.504730    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj\" (UniqueName: \"kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj\") pod \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") "
Nov 03 14:30:40.516636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.516599    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca\") pod \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") "
Nov 03 14:30:40.524256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.516867    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca\") pod \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\" (UID: \"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2\") "
Nov 03 14:30:40.548751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.531426    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca" pod "bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" (UID: "bbb011c3-b2ba-408a-bd16-2cc8497b0ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:40.548972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.548930    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca" pod "bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" (UID: "bbb011c3-b2ba-408a-bd16-2cc8497b0ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:30:40.550538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.550509    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj" pod "bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" (UID: "bbb011c3-b2ba-408a-bd16-2cc8497b0ca2"). InnerVolumeSpecName "kube-api-access-pqlmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:40.638831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.638801    6481 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 14:30:40.639002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.638848    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 14:30:40.649147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.649114    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlmj\" (UniqueName: \"kubernetes.io/projected/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-kube-api-access-pqlmj\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:40.649147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.649144    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:40.649334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.649156    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:40.768336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.768217    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:40.983298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:40.983261    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:41.188567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.188532    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"
Nov 03 14:30:41.188567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.188559    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp" event={"ID":"bbb011c3-b2ba-408a-bd16-2cc8497b0ca2","Type":"ContainerDied","Data":"65430d0c3f382e12963bc74a9a364042a4aea5dc73a87e796ccf003f7ce8e6f3"}
Nov 03 14:30:41.188791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.188604    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65430d0c3f382e12963bc74a9a364042a4aea5dc73a87e796ccf003f7ce8e6f3"
Nov 03 14:30:41.221926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.220356    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:41.638845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.638755    6481 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 14:30:41.638845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.638812    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 14:30:41.959650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:41.959616    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:42.171242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:42.171194    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:42.439124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:42.439084    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:42.654273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:42.648805    6481 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 14:30:42.654273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:42.648864    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 14:30:42.745406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:42.745334    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:30:42.836207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:42.836170    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"]
Nov 03 14:30:43.156158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.156067    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:43.230640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.230608    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"]
Nov 03 14:30:43.234296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.234272    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d608daf1-mlhf6"]
Nov 03 14:30:43.360260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.358825    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:43.576036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.576003    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:43.648836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.648796    6481 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 14:30:43.649047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:43.648871    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 14:30:44.067611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:44.067575    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf" path="/var/lib/kubelet/pods/d3f27aea-16bb-48c4-b3ec-1ee4d351ebaf/volumes"
Nov 03 14:30:44.159303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:44.159275    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:44.654251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:44.645542    6481 patch_prober.go:29] interesting pod/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Nov 03 14:30:44.654251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:44.645603    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Nov 03 14:30:44.713953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:44.713913    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:44.998962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:44.998924    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:45.614348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:45.613415    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:45.648002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:45.647967    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:45.648517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:45.648493    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:45.649585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:45.649557    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:30:47.811692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:47.811660    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:47.836623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:47.836575    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l" podUID="a4bdf93f-f1fa-4870-9d37-b56f812cfd04" containerName="payload-processing" containerID="cri-o://a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8" gracePeriod=25
Nov 03 14:30:48.026107 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.026034    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:48.115864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.115839    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:30:48.195893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.195860    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume\") pod \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") "
Nov 03 14:30:48.196103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.195913    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials\") pod \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") "
Nov 03 14:30:48.196103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.195973    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume\") pod \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") "
Nov 03 14:30:48.196103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.196054    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz\") pod \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\" (UID: \"a4bdf93f-f1fa-4870-9d37-b56f812cfd04\") "
Nov 03 14:30:48.196377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.196346    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume" pod "a4bdf93f-f1fa-4870-9d37-b56f812cfd04" (UID: "a4bdf93f-f1fa-4870-9d37-b56f812cfd04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:48.196513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.196402    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume" pod "a4bdf93f-f1fa-4870-9d37-b56f812cfd04" (UID: "a4bdf93f-f1fa-4870-9d37-b56f812cfd04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:48.196721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.196704    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:48.196789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.196724    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:48.198108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.198083    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials" pod "a4bdf93f-f1fa-4870-9d37-b56f812cfd04" (UID: "a4bdf93f-f1fa-4870-9d37-b56f812cfd04"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:48.198186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.198134    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz" pod "a4bdf93f-f1fa-4870-9d37-b56f812cfd04" (UID: "a4bdf93f-f1fa-4870-9d37-b56f812cfd04"). InnerVolumeSpecName "kube-api-access-rhqrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:48.278026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.277995    6481 generic.go:356] "Generic (PLEG): container finished" podID="a4bdf93f-f1fa-4870-9d37-b56f812cfd04" containerID="a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8" exitCode=0
Nov 03 14:30:48.278186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.278043    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"
Nov 03 14:30:48.278186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.278054    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l" event={"ID":"a4bdf93f-f1fa-4870-9d37-b56f812cfd04","Type":"ContainerDied","Data":"a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8"}
Nov 03 14:30:48.278186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.278114    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l" event={"ID":"a4bdf93f-f1fa-4870-9d37-b56f812cfd04","Type":"ContainerDied","Data":"acf5d1aa8e596bf5662f854e9dc195575dfa1c9fa441af98a58189219acf74e5"}
Nov 03 14:30:48.278186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.278145    6481 scope.go:122] "RemoveContainer" containerID="a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8"
Nov 03 14:30:48.289873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.289854    6481 scope.go:122] "RemoveContainer" containerID="a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8"
Nov 03 14:30:48.290148 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:48.290129    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8\": container with ID starting with a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8 not found: ID does not exist" containerID="a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8"
Nov 03 14:30:48.290189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.290156    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8"} err="failed to get container status \"a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8\": rpc error: code = NotFound desc = could not find container \"a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8\": container with ID starting with a677a98b2ad6df11fb71a1088785da153ed88e3034fe5a76fc03764d2b4ff0b8 not found: ID does not exist"
Nov 03 14:30:48.297796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.297775    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:48.297796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.297797    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhqrz\" (UniqueName: \"kubernetes.io/projected/a4bdf93f-f1fa-4870-9d37-b56f812cfd04-kube-api-access-rhqrz\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:48.332663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.332641    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"]
Nov 03 14:30:48.338260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:48.338200    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-6b542t72l"]
Nov 03 14:30:50.069023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:50.068969    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4bdf93f-f1fa-4870-9d37-b56f812cfd04" path="/var/lib/kubelet/pods/a4bdf93f-f1fa-4870-9d37-b56f812cfd04/volumes"
Nov 03 14:30:50.084157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:50.084123    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:50.305387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:50.305355    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:50.561779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:50.561745    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:51.043576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.043526    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"]
Nov 03 14:30:51.054711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.054685    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f99f44f-qpncc"]
Nov 03 14:30:51.487595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.487560    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"]
Nov 03 14:30:51.488151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.488136    6481 cpu_manager.go:414] "containerMap: removing container" podUID="bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" containerName="revoke-keys"
Nov 03 14:30:51.488200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.488151    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" containerName="revoke-keys"
Nov 03 14:30:51.488200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.488174    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a4bdf93f-f1fa-4870-9d37-b56f812cfd04" containerName="payload-processing"
Nov 03 14:30:51.488200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.488179    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4bdf93f-f1fa-4870-9d37-b56f812cfd04" containerName="payload-processing"
Nov 03 14:30:51.488325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.488318    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" containerName="revoke-keys"
Nov 03 14:30:51.488357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.488331    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4bdf93f-f1fa-4870-9d37-b56f812cfd04" containerName="payload-processing"
Nov 03 14:30:51.492010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.491996    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.511255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.511180    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:30:51.513897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.513876    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"]
Nov 03 14:30:51.520935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.520912    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:30:51.520935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.520931    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:30:51.521158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.521110    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:30:51.552203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.552167    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.552385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.552216    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.552385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.552349    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8qx\" (UniqueName: \"kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.653359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.653324    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.653359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.653358    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.653616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.653394    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8qx\" (UniqueName: \"kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.654015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.653991    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.655695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.655671    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.674862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.674836    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8qx\" (UniqueName: \"kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx\") pod \"maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:51.805967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:51.805899    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:52.071876 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:52.071807    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="188f74e6-a0b8-4790-b0a2-0593f0f9c421" path="/var/lib/kubelet/pods/188f74e6-a0b8-4790-b0a2-0593f0f9c421/volumes"
Nov 03 14:30:52.110776 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:30:52.110738    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d864f6_6d08_4548_95a0_be0f58d739ec.slice/crio-d5afe57cb4c2ccc42a76cf9ecdb08d416d3803a4ea0a73d9e5eed6d1c2a55b85 WatchSource:0}: Error finding container d5afe57cb4c2ccc42a76cf9ecdb08d416d3803a4ea0a73d9e5eed6d1c2a55b85: Status 404 returned error can't find the container with id d5afe57cb4c2ccc42a76cf9ecdb08d416d3803a4ea0a73d9e5eed6d1c2a55b85
Nov 03 14:30:52.111010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:52.110990    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"]
Nov 03 14:30:52.320592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:52.320526    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:52.331058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:52.331006    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp" event={"ID":"d4d864f6-6d08-4548-95a0-be0f58d739ec","Type":"ContainerStarted","Data":"d5afe57cb4c2ccc42a76cf9ecdb08d416d3803a4ea0a73d9e5eed6d1c2a55b85"}
Nov 03 14:30:53.648315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:53.648084    6481 generic.go:356] "Generic (PLEG): container finished" podID="d4d864f6-6d08-4548-95a0-be0f58d739ec" containerID="041471fac8e2ba11030445317a97036a151ffda1c196b670ed69b0e8bf9f6ef5" exitCode=0
Nov 03 14:30:53.648797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:53.648312    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp" event={"ID":"d4d864f6-6d08-4548-95a0-be0f58d739ec","Type":"ContainerDied","Data":"041471fac8e2ba11030445317a97036a151ffda1c196b670ed69b0e8bf9f6ef5"}
Nov 03 14:30:55.703255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.674795    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:55.853680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.853651    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:55.992203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.992089    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca\") pod \"d4d864f6-6d08-4548-95a0-be0f58d739ec\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") "
Nov 03 14:30:55.992418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.992317    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx\" (UniqueName: \"kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx\") pod \"d4d864f6-6d08-4548-95a0-be0f58d739ec\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") "
Nov 03 14:30:55.992488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.992457    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca\") pod \"d4d864f6-6d08-4548-95a0-be0f58d739ec\" (UID: \"d4d864f6-6d08-4548-95a0-be0f58d739ec\") "
Nov 03 14:30:55.992760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.992720    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca" pod "d4d864f6-6d08-4548-95a0-be0f58d739ec" (UID: "d4d864f6-6d08-4548-95a0-be0f58d739ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:30:55.992893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.992824    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:55.994974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.994943    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca" pod "d4d864f6-6d08-4548-95a0-be0f58d739ec" (UID: "d4d864f6-6d08-4548-95a0-be0f58d739ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:30:55.995306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:55.995273    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx" pod "d4d864f6-6d08-4548-95a0-be0f58d739ec" (UID: "d4d864f6-6d08-4548-95a0-be0f58d739ec"). InnerVolumeSpecName "kube-api-access-fk8qx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:30:56.094002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:56.093958    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8qx\" (UniqueName: \"kubernetes.io/projected/d4d864f6-6d08-4548-95a0-be0f58d739ec-kube-api-access-fk8qx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:56.094002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:56.093998    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4d864f6-6d08-4548-95a0-be0f58d739ec-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:30:56.109732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:56.109700    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"]
Nov 03 14:30:56.734160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:56.734131    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"
Nov 03 14:30:56.734160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:56.734147    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp" event={"ID":"d4d864f6-6d08-4548-95a0-be0f58d739ec","Type":"ContainerDied","Data":"d5afe57cb4c2ccc42a76cf9ecdb08d416d3803a4ea0a73d9e5eed6d1c2a55b85"}
Nov 03 14:30:56.734621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:56.734200    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5afe57cb4c2ccc42a76cf9ecdb08d416d3803a4ea0a73d9e5eed6d1c2a55b85"
Nov 03 14:30:57.687467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:57.687431    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:57.985781 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:30:57.985692    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:30:58.021490 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:58.021446    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:30:59.766886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:30:59.766842    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:01.110330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.110290    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m" podUID="b67feb46-a095-4ea5-97e6-8d60f999bce1" containerName="payload-processing" containerID="cri-o://641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d" gracePeriod=25
Nov 03 14:31:01.378747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.378690    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:31:01.457947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.457919    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume\") pod \"b67feb46-a095-4ea5-97e6-8d60f999bce1\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") "
Nov 03 14:31:01.458130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458013    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf\") pod \"b67feb46-a095-4ea5-97e6-8d60f999bce1\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") "
Nov 03 14:31:01.458130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458084    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials\") pod \"b67feb46-a095-4ea5-97e6-8d60f999bce1\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") "
Nov 03 14:31:01.458130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458122    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume\") pod \"b67feb46-a095-4ea5-97e6-8d60f999bce1\" (UID: \"b67feb46-a095-4ea5-97e6-8d60f999bce1\") "
Nov 03 14:31:01.458498 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458410    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume" pod "b67feb46-a095-4ea5-97e6-8d60f999bce1" (UID: "b67feb46-a095-4ea5-97e6-8d60f999bce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:01.458669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458573    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume" pod "b67feb46-a095-4ea5-97e6-8d60f999bce1" (UID: "b67feb46-a095-4ea5-97e6-8d60f999bce1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:01.458961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458940    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:01.459049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.458967    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b67feb46-a095-4ea5-97e6-8d60f999bce1-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:01.460258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.460221    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials" pod "b67feb46-a095-4ea5-97e6-8d60f999bce1" (UID: "b67feb46-a095-4ea5-97e6-8d60f999bce1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:01.460338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.460275    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf" pod "b67feb46-a095-4ea5-97e6-8d60f999bce1" (UID: "b67feb46-a095-4ea5-97e6-8d60f999bce1"). InnerVolumeSpecName "kube-api-access-t7khf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:01.560372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.560340    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7khf\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-kube-api-access-t7khf\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:01.560372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.560369    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b67feb46-a095-4ea5-97e6-8d60f999bce1-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:01.773688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.773648    6481 generic.go:356] "Generic (PLEG): container finished" podID="b67feb46-a095-4ea5-97e6-8d60f999bce1" containerID="641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d" exitCode=0
Nov 03 14:31:01.773879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.773706    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"
Nov 03 14:31:01.773879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.773740    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m" event={"ID":"b67feb46-a095-4ea5-97e6-8d60f999bce1","Type":"ContainerDied","Data":"641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d"}
Nov 03 14:31:01.773879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.773816    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m" event={"ID":"b67feb46-a095-4ea5-97e6-8d60f999bce1","Type":"ContainerDied","Data":"d772f8c25039c2495c3051d0edbfe4189824d25091eb35bf70b5959918aed1ca"}
Nov 03 14:31:01.773879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.773845    6481 scope.go:122] "RemoveContainer" containerID="641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d"
Nov 03 14:31:01.784444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.784426    6481 scope.go:122] "RemoveContainer" containerID="641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d"
Nov 03 14:31:01.784712 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:31:01.784692    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d\": container with ID starting with 641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d not found: ID does not exist" containerID="641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d"
Nov 03 14:31:01.784760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.784719    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d"} err="failed to get container status \"641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d\": rpc error: code = NotFound desc = could not find container \"641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d\": container with ID starting with 641f1a82baaf8afbe3aeb05f54d7cc03a8b353bc09ba63d9dfed7655220fb84d not found: ID does not exist"
Nov 03 14:31:01.818724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.818702    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:01.828260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.828239    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"]
Nov 03 14:31:01.840284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:01.840260    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc5gjp6m"]
Nov 03 14:31:02.068368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:02.068298    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67feb46-a095-4ea5-97e6-8d60f999bce1" path="/var/lib/kubelet/pods/b67feb46-a095-4ea5-97e6-8d60f999bce1/volumes"
Nov 03 14:31:02.106909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:02.106882    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"]
Nov 03 14:31:02.305564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:02.305528    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:02.514532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:02.514499    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:03.225183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.225146    6481 scope.go:122] "RemoveContainer" containerID="c596ac848ff6281ba781fa90fcaf48924752917b94743d5c26994bb3afcc2378"
Nov 03 14:31:03.243808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.243471    6481 scope.go:122] "RemoveContainer" containerID="e39bd9cb789481dc91f159dac65bc1d4b8eb11be5c79265cf170d36be938ff4b"
Nov 03 14:31:03.262204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.262179    6481 scope.go:122] "RemoveContainer" containerID="f04a6136464f23ef625c3a1097f737ecee5000a2d40308e84374ed04c5cd5980"
Nov 03 14:31:03.272638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.272618    6481 scope.go:122] "RemoveContainer" containerID="3d26fbc6f0147b9c25b62e43c07876ce5d67e4fc23bbf59ff9ea8dc3a4d2bdaa"
Nov 03 14:31:03.283376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.283359    6481 scope.go:122] "RemoveContainer" containerID="4e84a9653f9106570b38ada542ac63604d6562962b079151514031574b40ee53"
Nov 03 14:31:03.291931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.291910    6481 scope.go:122] "RemoveContainer" containerID="1b3c0e559857fb7800ff6ab386611f653a258f11652801668d0631521ac67305"
Nov 03 14:31:03.882274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:03.880603    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:05.039546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:05.039509    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"]
Nov 03 14:31:05.052441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:05.052410    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e212dc-fqwmk"]
Nov 03 14:31:05.283393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:05.283356    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:31:05.792378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:05.792348    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:06.024284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:06.009984    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:06.069849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:06.069749    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efceb72-b62f-43ec-9e84-3c42f667746e" path="/var/lib/kubelet/pods/8efceb72-b62f-43ec-9e84-3c42f667746e/volumes"
Nov 03 14:31:07.117249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:07.116430    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerName="payload-pre-processing" containerID="cri-o://5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be" gracePeriod=25
Nov 03 14:31:08.487908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:08.487866    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:10.483056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:10.483020    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:11.683495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.683466    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"]
Nov 03 14:31:11.684042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.684028    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d4d864f6-6d08-4548-95a0-be0f58d739ec" containerName="revoke-keys"
Nov 03 14:31:11.684091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.684043    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d864f6-6d08-4548-95a0-be0f58d739ec" containerName="revoke-keys"
Nov 03 14:31:11.684091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.684060    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b67feb46-a095-4ea5-97e6-8d60f999bce1" containerName="payload-processing"
Nov 03 14:31:11.684091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.684066    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67feb46-a095-4ea5-97e6-8d60f999bce1" containerName="payload-processing"
Nov 03 14:31:11.684182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.684174    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d864f6-6d08-4548-95a0-be0f58d739ec" containerName="revoke-keys"
Nov 03 14:31:11.684217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.684182    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67feb46-a095-4ea5-97e6-8d60f999bce1" containerName="payload-processing"
Nov 03 14:31:11.690036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.690018    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.715021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.714994    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"]
Nov 03 14:31:11.720951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.720931    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:31:11.721099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.721078    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:31:11.721152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.721122    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:31:11.721152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.721134    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:31:11.802154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.802120    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmwq\" (UniqueName: \"kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.802331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.802310    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.802376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.802344    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.903290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.903243    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmwq\" (UniqueName: \"kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.903459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.903417    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.903503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.903463    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.904139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.904113    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.905694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.905674    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:11.923816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:11.923796    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmwq\" (UniqueName: \"kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq\") pod \"maas-api-revoke-sub-71f4382e-4bfp5\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:12.004647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:12.004572    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:12.308122 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:31:12.308088    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5712849_7163_4019_b6d0_ecfdfc96e46c.slice/crio-83e586cc894c35c1ab35ce71f60d78d73255088d20533b2fe7b2281d48261464 WatchSource:0}: Error finding container 83e586cc894c35c1ab35ce71f60d78d73255088d20533b2fe7b2281d48261464: Status 404 returned error can't find the container with id 83e586cc894c35c1ab35ce71f60d78d73255088d20533b2fe7b2281d48261464
Nov 03 14:31:12.308522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:12.308501    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"]
Nov 03 14:31:12.515166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:12.515129    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:12.974912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:12.974876    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5" event={"ID":"e5712849-7163-4019-b6d0-ecfdfc96e46c","Type":"ContainerStarted","Data":"8fdf54fee0fa4c060a8c2ea4300cf8d6f88f2e7fa6b0bb778bf64a4d6315b928"}
Nov 03 14:31:12.975344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:12.974957    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5" event={"ID":"e5712849-7163-4019-b6d0-ecfdfc96e46c","Type":"ContainerStarted","Data":"83e586cc894c35c1ab35ce71f60d78d73255088d20533b2fe7b2281d48261464"}
Nov 03 14:31:13.014290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:13.014211    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5" podStartSLOduration=2.014197108 podStartE2EDuration="2.014197108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:13.014170607 +0000 UTC m=+1151.064201790" watchObservedRunningTime="2026-11-03 14:31:13.014197108 +0000 UTC m=+1151.064228291"
Nov 03 14:31:13.984342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:13.984302    6481 generic.go:356] "Generic (PLEG): container finished" podID="e5712849-7163-4019-b6d0-ecfdfc96e46c" containerID="8fdf54fee0fa4c060a8c2ea4300cf8d6f88f2e7fa6b0bb778bf64a4d6315b928" exitCode=0
Nov 03 14:31:13.984824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:13.984484    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5" event={"ID":"e5712849-7163-4019-b6d0-ecfdfc96e46c","Type":"ContainerDied","Data":"8fdf54fee0fa4c060a8c2ea4300cf8d6f88f2e7fa6b0bb778bf64a4d6315b928"}
Nov 03 14:31:14.495157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:14.495104    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:14.719779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:14.719747    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:31:15.146555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.146531    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:15.345659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.345580    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq\" (UniqueName: \"kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq\") pod \"e5712849-7163-4019-b6d0-ecfdfc96e46c\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") "
Nov 03 14:31:15.345809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.345682    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca\") pod \"e5712849-7163-4019-b6d0-ecfdfc96e46c\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") "
Nov 03 14:31:15.345809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.345734    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca\") pod \"e5712849-7163-4019-b6d0-ecfdfc96e46c\" (UID: \"e5712849-7163-4019-b6d0-ecfdfc96e46c\") "
Nov 03 14:31:15.346126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.346084    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca" pod "e5712849-7163-4019-b6d0-ecfdfc96e46c" (UID: "e5712849-7163-4019-b6d0-ecfdfc96e46c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:15.347667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.347647    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq" pod "e5712849-7163-4019-b6d0-ecfdfc96e46c" (UID: "e5712849-7163-4019-b6d0-ecfdfc96e46c"). InnerVolumeSpecName "kube-api-access-ghmwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:15.347750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.347732    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca" pod "e5712849-7163-4019-b6d0-ecfdfc96e46c" (UID: "e5712849-7163-4019-b6d0-ecfdfc96e46c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:31:15.447495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.447466    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmwq\" (UniqueName: \"kubernetes.io/projected/e5712849-7163-4019-b6d0-ecfdfc96e46c-kube-api-access-ghmwq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:15.447495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.447491    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:15.447684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:15.447503    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5712849-7163-4019-b6d0-ecfdfc96e46c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:16.001693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:16.001666    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"
Nov 03 14:31:16.001693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:16.001673    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5" event={"ID":"e5712849-7163-4019-b6d0-ecfdfc96e46c","Type":"ContainerDied","Data":"83e586cc894c35c1ab35ce71f60d78d73255088d20533b2fe7b2281d48261464"}
Nov 03 14:31:16.001906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:16.001713    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e586cc894c35c1ab35ce71f60d78d73255088d20533b2fe7b2281d48261464"
Nov 03 14:31:16.590775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:16.590746    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:17.718844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.718802    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"]
Nov 03 14:31:17.719689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.719666    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e5712849-7163-4019-b6d0-ecfdfc96e46c" containerName="revoke-keys"
Nov 03 14:31:17.719689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.719688    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5712849-7163-4019-b6d0-ecfdfc96e46c" containerName="revoke-keys"
Nov 03 14:31:17.719855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.719826    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5712849-7163-4019-b6d0-ecfdfc96e46c" containerName="revoke-keys"
Nov 03 14:31:17.723414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.723394    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.751044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.751007    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"]
Nov 03 14:31:17.774379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774350    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774393    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774562    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774688    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7bx\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774743    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774772    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774890    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774930    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.774966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.774954    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.876565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876528    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.876789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876579    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.876789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876659    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.876789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876688    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.876789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876775    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876881    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7bx\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876943    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.876973    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.877011    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.877059    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.877273    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877399 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.877376    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.877447    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.877521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.877459    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.878924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.878903    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.879381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.879362    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.897371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.897347    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:17.897371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:17.897360    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7bx\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx\") pod \"e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") " pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:18.038698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.038606    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:18.043054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.043034    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"]
Nov 03 14:31:18.047180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.047157    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e127a61f-7mg8w"]
Nov 03 14:31:18.070976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.070949    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab1ed149-4286-4a8b-99f2-fafa27d52990" path="/var/lib/kubelet/pods/ab1ed149-4286-4a8b-99f2-fafa27d52990/volumes"
Nov 03 14:31:18.361551 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:31:18.361525    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a781268_b0a7_4b29_afbb_eb23570b25d8.slice/crio-4a578de4a87c34c328bdf64994a6c7e65ccb8259e550c81cea63e8045f0d877f WatchSource:0}: Error finding container 4a578de4a87c34c328bdf64994a6c7e65ccb8259e550c81cea63e8045f0d877f: Status 404 returned error can't find the container with id 4a578de4a87c34c328bdf64994a6c7e65ccb8259e550c81cea63e8045f0d877f
Nov 03 14:31:18.361867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.361846    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"]
Nov 03 14:31:18.363990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.363957    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:31:18.364077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.364021    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:31:18.364077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.364052    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:31:18.577909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:18.577870    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:19.033970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:19.033933    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" event={"ID":"5a781268-b0a7-4b29-afbb-eb23570b25d8","Type":"ContainerStarted","Data":"9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120"}
Nov 03 14:31:19.034385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:19.033993    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" event={"ID":"5a781268-b0a7-4b29-afbb-eb23570b25d8","Type":"ContainerStarted","Data":"4a578de4a87c34c328bdf64994a6c7e65ccb8259e550c81cea63e8045f0d877f"}
Nov 03 14:31:19.038863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:19.038842    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:19.040170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:19.040144    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:19.040247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:19.040199    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:19.093361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:19.093317    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podStartSLOduration=2.093304338 podStartE2EDuration="2.093304338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:19.092252138 +0000 UTC m=+1157.142283319" watchObservedRunningTime="2026-11-03 14:31:19.093304338 +0000 UTC m=+1157.143335521"
Nov 03 14:31:20.039581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.039541    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:20.039969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.039601    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:20.345211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.345127    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"]
Nov 03 14:31:20.349291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.349268    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.372643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.372615    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"]
Nov 03 14:31:20.378545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.378517    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:31:20.378709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.378599    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:31:20.378709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.378670    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:31:20.378709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.378685    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:31:20.551254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.550756    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.594295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.550989    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.594295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.585500    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcgxz\" (UniqueName: \"kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.711041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.710996    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.711390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.711364    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.711622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.711602    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcgxz\" (UniqueName: \"kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.712951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.712929    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.757253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.736802    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcgxz\" (UniqueName: \"kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.757253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.754773    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d4f417a-7w22m\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:20.966034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:20.965946    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:21.055148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:21.054646    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:21.055148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:21.054724    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:21.299728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:21.273192    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:21.721943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:21.721551    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"]
Nov 03 14:31:22.045256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:22.042637    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:22.045256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:22.042692    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:22.238287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:22.230030    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m" event={"ID":"5bd5027b-34d7-4f38-9bc6-f9c77d868292","Type":"ContainerStarted","Data":"67540fa28c09f322418424c1bfaa423e03bab263cd45330c8964f3580560ac79"}
Nov 03 14:31:23.042253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.041733    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:23.042253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.041797    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:23.239267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.236330    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:31:23.269382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.265331    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:31:23.314676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.313934    6481 generic.go:356] "Generic (PLEG): container finished" podID="5bd5027b-34d7-4f38-9bc6-f9c77d868292" containerID="bf2091cb6dadbded5ac03c705ba8f7bd20cbb665b9c0df00fe9770fcd8bf6133" exitCode=0
Nov 03 14:31:23.314676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.314130    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m" event={"ID":"5bd5027b-34d7-4f38-9bc6-f9c77d868292","Type":"ContainerDied","Data":"bf2091cb6dadbded5ac03c705ba8f7bd20cbb665b9c0df00fe9770fcd8bf6133"}
Nov 03 14:31:23.332446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.331408    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:31:23.389740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:23.389320    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:31:24.023254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.009104    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:24.039675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.039645    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:24.039813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.039697    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:24.556079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.556056    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:24.630195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.630109    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca\") pod \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") "
Nov 03 14:31:24.630384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.630353    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca\") pod \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") "
Nov 03 14:31:24.630555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.630533    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz\" (UniqueName: \"kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz\") pod \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\" (UID: \"5bd5027b-34d7-4f38-9bc6-f9c77d868292\") "
Nov 03 14:31:24.630688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.630660    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca" pod "5bd5027b-34d7-4f38-9bc6-f9c77d868292" (UID: "5bd5027b-34d7-4f38-9bc6-f9c77d868292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:24.631085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.631054    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:24.633010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.632982    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz" pod "5bd5027b-34d7-4f38-9bc6-f9c77d868292" (UID: "5bd5027b-34d7-4f38-9bc6-f9c77d868292"). InnerVolumeSpecName "kube-api-access-tcgxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:24.633198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.633172    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca" pod "5bd5027b-34d7-4f38-9bc6-f9c77d868292" (UID: "5bd5027b-34d7-4f38-9bc6-f9c77d868292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:31:24.725692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.725651    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:31:24.732408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.732381    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcgxz\" (UniqueName: \"kubernetes.io/projected/5bd5027b-34d7-4f38-9bc6-f9c77d868292-kube-api-access-tcgxz\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:24.732408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:24.732406    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bd5027b-34d7-4f38-9bc6-f9c77d868292-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:25.039273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:25.039219    6481 patch_prober.go:29] interesting pod/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Nov 03 14:31:25.039454 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:25.039290    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Nov 03 14:31:25.341948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:25.341867    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"
Nov 03 14:31:25.342096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:25.341875    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m" event={"ID":"5bd5027b-34d7-4f38-9bc6-f9c77d868292","Type":"ContainerDied","Data":"67540fa28c09f322418424c1bfaa423e03bab263cd45330c8964f3580560ac79"}
Nov 03 14:31:25.342096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:25.342000    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67540fa28c09f322418424c1bfaa423e03bab263cd45330c8964f3580560ac79"
Nov 03 14:31:26.042826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:26.042799    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:26.043351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:26.043336    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:26.044022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:26.043999    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:31:26.345207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:26.345115    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:27.326050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.326029    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:31:27.357430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.357402    6481 generic.go:356] "Generic (PLEG): container finished" podID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerID="5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be" exitCode=0
Nov 03 14:31:27.357545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.357439    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"
Nov 03 14:31:27.357545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.357474    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" event={"ID":"169973d6-32ab-4a5a-ba21-7310d53d8493","Type":"ContainerDied","Data":"5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be"}
Nov 03 14:31:27.357545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.357523    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn" event={"ID":"169973d6-32ab-4a5a-ba21-7310d53d8493","Type":"ContainerDied","Data":"0ec393920fc173dc862818c6b03d253ad8088bd405b179790f3875ff9afc31ae"}
Nov 03 14:31:27.357707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.357547    6481 scope.go:122] "RemoveContainer" containerID="5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be"
Nov 03 14:31:27.367715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.367690    6481 scope.go:122] "RemoveContainer" containerID="5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be"
Nov 03 14:31:27.368016 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:31:27.367983    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be\": container with ID starting with 5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be not found: ID does not exist" containerID="5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be"
Nov 03 14:31:27.368095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.368016    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be"} err="failed to get container status \"5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be\": rpc error: code = NotFound desc = could not find container \"5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be\": container with ID starting with 5ce085ba27585a4ef5258d890ed68c15f0b3ee16567361c4b0545d967edac2be not found: ID does not exist"
Nov 03 14:31:27.482788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.482761    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c\" (UniqueName: \"kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c\") pod \"169973d6-32ab-4a5a-ba21-7310d53d8493\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") "
Nov 03 14:31:27.482935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.482849    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume\") pod \"169973d6-32ab-4a5a-ba21-7310d53d8493\" (UID: \"169973d6-32ab-4a5a-ba21-7310d53d8493\") "
Nov 03 14:31:27.483201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.483179    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume" pod "169973d6-32ab-4a5a-ba21-7310d53d8493" (UID: "169973d6-32ab-4a5a-ba21-7310d53d8493"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:27.483579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.483555    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169973d6-32ab-4a5a-ba21-7310d53d8493-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:27.484945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.484923    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c" pod "169973d6-32ab-4a5a-ba21-7310d53d8493" (UID: "169973d6-32ab-4a5a-ba21-7310d53d8493"). InnerVolumeSpecName "kube-api-access-wmq2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:27.584627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.584603    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmq2c\" (UniqueName: \"kubernetes.io/projected/169973d6-32ab-4a5a-ba21-7310d53d8493-kube-api-access-wmq2c\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:27.706379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.706347    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"]
Nov 03 14:31:27.717826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:27.717803    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4a67e25f-d9b789fc9-ss6wn"]
Nov 03 14:31:27.986592 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:31:27.986561    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:31:28.044841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.044806    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"]
Nov 03 14:31:28.051004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.050971    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa080eb0-2tdt6"]
Nov 03 14:31:28.075175 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.075145    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" path="/var/lib/kubelet/pods/169973d6-32ab-4a5a-ba21-7310d53d8493/volumes"
Nov 03 14:31:28.075638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.075623    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffdb16d-6147-4da6-b43f-ddea11bd00ec" path="/var/lib/kubelet/pods/6ffdb16d-6147-4da6-b43f-ddea11bd00ec/volumes"
Nov 03 14:31:28.533123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.533071    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:28.705853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.705815    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"]
Nov 03 14:31:28.706459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.706443    6481 cpu_manager.go:414] "containerMap: removing container" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerName="payload-pre-processing"
Nov 03 14:31:28.706513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.706459    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerName="payload-pre-processing"
Nov 03 14:31:28.706513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.706478    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5bd5027b-34d7-4f38-9bc6-f9c77d868292" containerName="revoke-keys"
Nov 03 14:31:28.706513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.706484    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd5027b-34d7-4f38-9bc6-f9c77d868292" containerName="revoke-keys"
Nov 03 14:31:28.706613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.706571    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="169973d6-32ab-4a5a-ba21-7310d53d8493" containerName="payload-pre-processing"
Nov 03 14:31:28.706613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.706580    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd5027b-34d7-4f38-9bc6-f9c77d868292" containerName="revoke-keys"
Nov 03 14:31:28.710812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.710792    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:28.738000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.737971    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:31:28.738177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.738109    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:31:28.738177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.738160    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:31:28.738341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.738218    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:31:28.740454 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.740425    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"]
Nov 03 14:31:28.903134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.903041    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:28.903134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.903088    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2h7\" (UniqueName: \"kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:28.903357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:28.903335    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.004449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.004414    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.004624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.004493    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.004624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.004515    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2h7\" (UniqueName: \"kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.005048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.005025    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.006928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.006900    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.024867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.024842    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2h7\" (UniqueName: \"kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7\") pod \"maas-api-revoke-sub-02f28ce9-d5c9m\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.028704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.028678    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:29.337559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.337532    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"]
Nov 03 14:31:29.337927 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:31:29.337903    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b4d115_e5bc_4231_837e_7e23ba26f435.slice/crio-daa14b1fc4283d95b1b775d6b2f049160d552db459b0d8f5db381c3adbc1dbf2 WatchSource:0}: Error finding container daa14b1fc4283d95b1b775d6b2f049160d552db459b0d8f5db381c3adbc1dbf2: Status 404 returned error can't find the container with id daa14b1fc4283d95b1b775d6b2f049160d552db459b0d8f5db381c3adbc1dbf2
Nov 03 14:31:29.374209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:29.374184    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m" event={"ID":"f9b4d115-e5bc-4231-837e-7e23ba26f435","Type":"ContainerStarted","Data":"daa14b1fc4283d95b1b775d6b2f049160d552db459b0d8f5db381c3adbc1dbf2"}
Nov 03 14:31:30.383434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:30.383350    6481 generic.go:356] "Generic (PLEG): container finished" podID="f9b4d115-e5bc-4231-837e-7e23ba26f435" containerID="bff5997da6f2a691cf51f00c9f34321feee923925eaefa93a6b2d97d21c27603" exitCode=0
Nov 03 14:31:30.383805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:30.383437    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m" event={"ID":"f9b4d115-e5bc-4231-837e-7e23ba26f435","Type":"ContainerDied","Data":"bff5997da6f2a691cf51f00c9f34321feee923925eaefa93a6b2d97d21c27603"}
Nov 03 14:31:30.675453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:30.675422    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:31.282480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.282450    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:31:31.546474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.546412    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:31.639761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.639727    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca\") pod \"f9b4d115-e5bc-4231-837e-7e23ba26f435\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") "
Nov 03 14:31:31.639962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.639782    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca\") pod \"f9b4d115-e5bc-4231-837e-7e23ba26f435\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") "
Nov 03 14:31:31.640669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.640643    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca" pod "f9b4d115-e5bc-4231-837e-7e23ba26f435" (UID: "f9b4d115-e5bc-4231-837e-7e23ba26f435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:31.640877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.639874    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7\" (UniqueName: \"kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7\") pod \"f9b4d115-e5bc-4231-837e-7e23ba26f435\" (UID: \"f9b4d115-e5bc-4231-837e-7e23ba26f435\") "
Nov 03 14:31:31.641425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.641404    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:31.644139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.642806    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca" pod "f9b4d115-e5bc-4231-837e-7e23ba26f435" (UID: "f9b4d115-e5bc-4231-837e-7e23ba26f435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:31:31.644345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.644324    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7" pod "f9b4d115-e5bc-4231-837e-7e23ba26f435" (UID: "f9b4d115-e5bc-4231-837e-7e23ba26f435"). InnerVolumeSpecName "kube-api-access-cw2h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:31.744099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.744061    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b4d115-e5bc-4231-837e-7e23ba26f435-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:31.744099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:31.744091    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw2h7\" (UniqueName: \"kubernetes.io/projected/f9b4d115-e5bc-4231-837e-7e23ba26f435-kube-api-access-cw2h7\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:32.405019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:32.404921    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m" event={"ID":"f9b4d115-e5bc-4231-837e-7e23ba26f435","Type":"ContainerDied","Data":"daa14b1fc4283d95b1b775d6b2f049160d552db459b0d8f5db381c3adbc1dbf2"}
Nov 03 14:31:32.405202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:32.405023    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daa14b1fc4283d95b1b775d6b2f049160d552db459b0d8f5db381c3adbc1dbf2"
Nov 03 14:31:32.405202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:32.405094    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"
Nov 03 14:31:33.753344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:33.753309    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:36.093714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:36.093648    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:38.206833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:38.206779    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:40.233923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:40.233888    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:40.447946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:40.447892    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:41.045823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:41.045789    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"]
Nov 03 14:31:41.057101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:41.057072    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17522a21-vzt24"]
Nov 03 14:31:42.067889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:42.067857    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5111e93b-6a39-49ea-a56b-fa243e84011a" path="/var/lib/kubelet/pods/5111e93b-6a39-49ea-a56b-fa243e84011a/volumes"
Nov 03 14:31:42.121925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:42.121897    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:42.632620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:42.632582    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:42.864274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:42.864194    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:43.191187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.191108    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:43.413736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.413702    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:43.492160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.492060    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"]
Nov 03 14:31:43.499796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.499768    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4a67e25f-bfcbp"]
Nov 03 14:31:43.626617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.626587    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:43.694797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.694767    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"]
Nov 03 14:31:43.695389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.695374    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f9b4d115-e5bc-4231-837e-7e23ba26f435" containerName="revoke-keys"
Nov 03 14:31:43.695452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.695390    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b4d115-e5bc-4231-837e-7e23ba26f435" containerName="revoke-keys"
Nov 03 14:31:43.695517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.695508    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b4d115-e5bc-4231-837e-7e23ba26f435" containerName="revoke-keys"
Nov 03 14:31:43.700144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.700123    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.716852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.716832    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:31:43.716955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.716916    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:31:43.720359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.720338    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"]
Nov 03 14:31:43.734726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.734704    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:31:43.735014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.734993    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:31:43.811480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.811426    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.811480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.811475    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsj6d\" (UniqueName: \"kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.811648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.811626    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.844636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.844616    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:43.913273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.913216    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsj6d\" (UniqueName: \"kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.913382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.913367    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.913508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.913494    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.913981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.913956    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.915727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.915709    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:43.934147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:43.934123    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsj6d\" (UniqueName: \"kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d\") pod \"maas-api-revoke-sub-1c471679-lr88j\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:44.014213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:44.014181    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:44.069927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:44.069901    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d864f6-6d08-4548-95a0-be0f58d739ec" path="/var/lib/kubelet/pods/d4d864f6-6d08-4548-95a0-be0f58d739ec/volumes"
Nov 03 14:31:44.334952 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:31:44.334919    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c19cb1a_a4af_4ed7_b09d_e76b6cd7c324.slice/crio-f59ae7acb57a1ed35ade575c2c994952874f542ceb453b6f93d3beb38c56afef WatchSource:0}: Error finding container f59ae7acb57a1ed35ade575c2c994952874f542ceb453b6f93d3beb38c56afef: Status 404 returned error can't find the container with id f59ae7acb57a1ed35ade575c2c994952874f542ceb453b6f93d3beb38c56afef
Nov 03 14:31:44.335323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:44.335298    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"]
Nov 03 14:31:44.537156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:44.537112    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j" event={"ID":"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324","Type":"ContainerStarted","Data":"f59ae7acb57a1ed35ade575c2c994952874f542ceb453b6f93d3beb38c56afef"}
Nov 03 14:31:44.964755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:44.964728    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:45.228915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:45.228843    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:45.550544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:45.550464    6481 generic.go:356] "Generic (PLEG): container finished" podID="0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" containerID="0c772314efa05fafeb2e46c60a9be4e8900e8c8f9955e95ae2b7c173fcf0f10f" exitCode=0
Nov 03 14:31:45.550991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:45.550545    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j" event={"ID":"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324","Type":"ContainerDied","Data":"0c772314efa05fafeb2e46c60a9be4e8900e8c8f9955e95ae2b7c173fcf0f10f"}
Nov 03 14:31:45.713434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:45.713385    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:46.023843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:46.023811    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:46.070251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:46.069784    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"]
Nov 03 14:31:46.151249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:46.143767    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1c71c5c-4zlvk"]
Nov 03 14:31:46.286722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:46.286594    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:46.556704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:46.556622    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:47.122814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.122771    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:47.199254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.195753    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca\") pod \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") "
Nov 03 14:31:47.199254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.195820    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca\") pod \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") "
Nov 03 14:31:47.199254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.195894    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d\" (UniqueName: \"kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d\") pod \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\" (UID: \"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324\") "
Nov 03 14:31:47.200299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.200272    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca" pod "0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" (UID: "0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:31:47.202517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.202492    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:47.241376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.241342    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca" pod "0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" (UID: "0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:31:47.304690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.304592    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:47.334454 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.334419    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d" pod "0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" (UID: "0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324"). InnerVolumeSpecName "kube-api-access-nsj6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:31:47.408847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.405890    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsj6d\" (UniqueName: \"kubernetes.io/projected/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324-kube-api-access-nsj6d\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:31:47.607962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.607876    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j" event={"ID":"0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324","Type":"ContainerDied","Data":"f59ae7acb57a1ed35ade575c2c994952874f542ceb453b6f93d3beb38c56afef"}
Nov 03 14:31:47.608519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.608500    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f59ae7acb57a1ed35ade575c2c994952874f542ceb453b6f93d3beb38c56afef"
Nov 03 14:31:47.608678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:47.608665    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"
Nov 03 14:31:48.125060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:48.125024    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731308c2-51f5-438c-9d15-ccb69799717a" path="/var/lib/kubelet/pods/731308c2-51f5-438c-9d15-ccb69799717a/volumes"
Nov 03 14:31:48.258251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:48.256797    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:48.872259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:48.826728    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:49.158709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:49.158676    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:49.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:49.369180    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:50.909807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:50.909760    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:51.125670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:51.125638    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:51.338368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:51.338282    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:51.897986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:51.897949    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:52.044907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:52.044873    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"]
Nov 03 14:31:52.055606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:52.055567    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e78d10-7trzv"]
Nov 03 14:31:52.067545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:52.067520    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a63e8e8-5fbd-4a9f-8020-072fdd5fb542" path="/var/lib/kubelet/pods/9a63e8e8-5fbd-4a9f-8020-072fdd5fb542/volumes"
Nov 03 14:31:52.112887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:52.112861    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:52.834625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:52.834591    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:53.041257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.037431    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"]
Nov 03 14:31:53.041257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.038556    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" containerName="revoke-keys"
Nov 03 14:31:53.041257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.038579    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" containerName="revoke-keys"
Nov 03 14:31:53.041257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.040666    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" containerName="revoke-keys"
Nov 03 14:31:53.099652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.099450    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:53.108504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.108470    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"]
Nov 03 14:31:53.109265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.108965    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.144585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.141201    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4a67e25f\""
Nov 03 14:31:53.144585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.141335    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4a67e25f-dockercfg-gt2pt\""
Nov 03 14:31:53.177664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.177632    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9f5p\" (UniqueName: \"kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.177844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.177804    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.278834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.278799    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9f5p\" (UniqueName: \"kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.279002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.278918    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.279609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.279590    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.299358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.299331    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9f5p\" (UniqueName: \"kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p\") pod \"payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") " pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.477177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.477147    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:53.681100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.681068    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:53.782633 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:31:53.782602    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3954c1a4_a4a7_48b3_a635_e9ebc415f25e.slice/crio-8ad32d1a255562a1c9c99232d52213cce786ba0ab39baa7c6000ca8051d6be1e WatchSource:0}: Error finding container 8ad32d1a255562a1c9c99232d52213cce786ba0ab39baa7c6000ca8051d6be1e: Status 404 returned error can't find the container with id 8ad32d1a255562a1c9c99232d52213cce786ba0ab39baa7c6000ca8051d6be1e
Nov 03 14:31:53.783139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.783052    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"]
Nov 03 14:31:53.796074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:53.796045    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq" event={"ID":"3954c1a4-a4a7-48b3-a635-e9ebc415f25e","Type":"ContainerStarted","Data":"8ad32d1a255562a1c9c99232d52213cce786ba0ab39baa7c6000ca8051d6be1e"}
Nov 03 14:31:54.804929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:54.804889    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq" event={"ID":"3954c1a4-a4a7-48b3-a635-e9ebc415f25e","Type":"ContainerStarted","Data":"8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a"}
Nov 03 14:31:54.805351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:54.804976    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:31:54.848539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:54.848494    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq" podStartSLOduration=2.848478862 podStartE2EDuration="2.848478862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:54.847099134 +0000 UTC m=+1192.897130341" watchObservedRunningTime="2026-11-03 14:31:54.848478862 +0000 UTC m=+1192.898510046"
Nov 03 14:31:54.892077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:54.892050    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:55.447310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:55.447271    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:55.750217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:55.750139    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:55.962214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:55.962181    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:57.052997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:57.052963    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"]
Nov 03 14:31:57.064651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:57.064618    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d34580-dvctc"]
Nov 03 14:31:57.985260 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:31:57.985206    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:31:58.068589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:58.068558    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7599554e-f0ee-43c3-ac31-5ca551d718ce" path="/var/lib/kubelet/pods/7599554e-f0ee-43c3-ac31-5ca551d718ce/volumes"
Nov 03 14:31:58.145196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:58.145168    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:58.354043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:58.353965    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:31:58.722259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:31:58.722207    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:01.000071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.000036    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"]
Nov 03 14:32:01.004503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.004484    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.034846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.034823    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"]
Nov 03 14:32:01.040867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.040820    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6653eb\""
Nov 03 14:32:01.040867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.040839    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6653eb-dockercfg-p79hx\""
Nov 03 14:32:01.098177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.098135    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.098693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.098665    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxqr5\" (UniqueName: \"kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5\") pod \"payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.223980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.223935    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxqr5\" (UniqueName: \"kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5\") pod \"payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.224176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.224028    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.226655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.226630    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.261332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.251922    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxqr5\" (UniqueName: \"kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5\") pod \"payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.317947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.317924    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.626714 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:01.626685    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1ff6e4_3d98_4d65_a051_3391bc23672a.slice/crio-5af5b628b40fc3d209b0ee8343aca36de898c61f362a6d2499b8035114c55da4 WatchSource:0}: Error finding container 5af5b628b40fc3d209b0ee8343aca36de898c61f362a6d2499b8035114c55da4: Status 404 returned error can't find the container with id 5af5b628b40fc3d209b0ee8343aca36de898c61f362a6d2499b8035114c55da4
Nov 03 14:32:01.628698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.628668    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"]
Nov 03 14:32:01.862525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.862487    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c" event={"ID":"3f1ff6e4-3d98-4d65-a051-3391bc23672a","Type":"ContainerStarted","Data":"304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0"}
Nov 03 14:32:01.862709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.862551    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c" event={"ID":"3f1ff6e4-3d98-4d65-a051-3391bc23672a","Type":"ContainerStarted","Data":"5af5b628b40fc3d209b0ee8343aca36de898c61f362a6d2499b8035114c55da4"}
Nov 03 14:32:01.862709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.862583    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:01.906961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:01.906918    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c" podStartSLOduration=1.906899289 podStartE2EDuration="1.906899289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:01.90533417 +0000 UTC m=+1199.955365352" watchObservedRunningTime="2026-11-03 14:32:01.906899289 +0000 UTC m=+1199.956930472"
Nov 03 14:32:02.007693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:02.007661    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:32:02.228203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:02.228131    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:02.444871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:02.444843    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:03.061254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:03.060755    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:03.429648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:03.429606    6481 scope.go:122] "RemoveContainer" containerID="40fdb28ec759a6817bff5c9c9770002b3888c348fac73913f6bf41a421a0eece"
Nov 03 14:32:03.504252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:03.501406    6481 scope.go:122] "RemoveContainer" containerID="2fd9e9eaeca62ee1027426e5c2f23307a22617929d9971b2264d49adf7e11d88"
Nov 03 14:32:03.778524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:03.774412    6481 scope.go:122] "RemoveContainer" containerID="392c0d4a287d08a2831536f3f796f545a650e6cc850d0d6823bfd206d75bfcd5"
Nov 03 14:32:04.019083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.018962    6481 scope.go:122] "RemoveContainer" containerID="0eb2f4fff5340baf6b5eb3bdca98f25436e530ce58f8cb335662f9af3a0da299"
Nov 03 14:32:04.132521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.132492    6481 scope.go:122] "RemoveContainer" containerID="39e09bf352b788b60271bc4fa05fcd87c1cf0554d9979531fe61ed0cf1b67380"
Nov 03 14:32:04.297308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.279637    6481 scope.go:122] "RemoveContainer" containerID="8760aba4c7ba47d354038b3844f2cf6be0123f20936963c2bc04db52eba8c917"
Nov 03 14:32:04.297308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.281708    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts" podUID="6212728d-6351-4e9b-bbfd-f9424c48412b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:32:04.679953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.679913    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:04.887132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.887054    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:04.944477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.944449    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"]
Nov 03 14:32:04.949919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.949899    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:04.966087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.966063    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:04.966293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.966091    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfq8\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:04.966293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.966112    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:04.966435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.966288    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:04.970891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.970867    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"]
Nov 03 14:32:04.978318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.978285    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-95ac97e0ca1e\""
Nov 03 14:32:04.978318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.978295    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-b28d34176fc3\""
Nov 03 14:32:04.978489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.978388    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-c88df00e610c\""
Nov 03 14:32:04.978849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.978829    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\""
Nov 03 14:32:04.978955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.978871    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-xvjqk\""
Nov 03 14:32:04.981433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.981412    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\""
Nov 03 14:32:04.981433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.981416    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\""
Nov 03 14:32:04.981582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:04.981441    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\""
Nov 03 14:32:05.067533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.067502    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.067687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.067538    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfq8\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.067687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.067559    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.067687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.067636    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.068197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.068166    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.068329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.068312    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.070180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.070159    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.088373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.088348    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfq8\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.263260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.263204    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:05.571734 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:05.571695    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18543d3_a77c_435f_acb9_78fee0d7cf2f.slice/crio-1114a6755ce0c67fd13bad4248ee1e8d0d791bac19c4bbe7c19cb557525532e1 WatchSource:0}: Error finding container 1114a6755ce0c67fd13bad4248ee1e8d0d791bac19c4bbe7c19cb557525532e1: Status 404 returned error can't find the container with id 1114a6755ce0c67fd13bad4248ee1e8d0d791bac19c4bbe7c19cb557525532e1
Nov 03 14:32:05.572410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.572318    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"]
Nov 03 14:32:05.639959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.639937    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:05.816489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:05.816462    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:32:06.185517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:06.185483    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt" event={"ID":"a18543d3-a77c-435f-acb9-78fee0d7cf2f","Type":"ContainerStarted","Data":"e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5"}
Nov 03 14:32:06.185759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:06.185541    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt" event={"ID":"a18543d3-a77c-435f-acb9-78fee0d7cf2f","Type":"ContainerStarted","Data":"1114a6755ce0c67fd13bad4248ee1e8d0d791bac19c4bbe7c19cb557525532e1"}
Nov 03 14:32:06.185759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:06.185608    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:06.233004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:06.232942    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt" podStartSLOduration=2.232924237 podStartE2EDuration="2.232924237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:06.226921658 +0000 UTC m=+1204.276952848" watchObservedRunningTime="2026-11-03 14:32:06.232924237 +0000 UTC m=+1204.282955421"
Nov 03 14:32:06.881671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:06.881644    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:07.046285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:07.046254    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"]
Nov 03 14:32:07.057779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:07.057751    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc0e052c-pdzz2"]
Nov 03 14:32:07.659994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:07.659955    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:07.879391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:07.879357    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:08.067475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:08.067401    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0923ce12-8c7e-43fb-9aee-0f1b26e4ea20" path="/var/lib/kubelet/pods/0923ce12-8c7e-43fb-9aee-0f1b26e4ea20/volumes"
Nov 03 14:32:09.791114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:09.791078    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:11.816923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:11.816896    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:12.907704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:12.907678    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:32:13.912316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:13.912285    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:14.415491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.415454    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:14.507831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.507790    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"]
Nov 03 14:32:14.515205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.515180    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.528544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.528521    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"]
Nov 03 14:32:14.534517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.534492    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:14.534648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.534556    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:14.534711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.534663    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:14.546121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.546098    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:14.609326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.609302    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4jc\" (UniqueName: \"kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.609459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.609448    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.609506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.609488    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.626584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.626558    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:14.711281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.711193    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.711281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.711275    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.711440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.711408    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4jc\" (UniqueName: \"kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.711979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.711961    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.713528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.713507    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.732338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.732311    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4jc\" (UniqueName: \"kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc\") pod \"maas-api-revoke-sub-5d378834-hdqjk\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:14.839631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:14.839601    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:15.149672 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:15.149642    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1347bde2_d8a0_48f0_9dc7_826ab61048f5.slice/crio-79410f74b7c1c9b57a9f80d7e9a0a86f7b04ed7e8939d8d417399bab9eb5b7ce WatchSource:0}: Error finding container 79410f74b7c1c9b57a9f80d7e9a0a86f7b04ed7e8939d8d417399bab9eb5b7ce: Status 404 returned error can't find the container with id 79410f74b7c1c9b57a9f80d7e9a0a86f7b04ed7e8939d8d417399bab9eb5b7ce
Nov 03 14:32:15.150753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:15.150727    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"]
Nov 03 14:32:15.253676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:15.253650    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk" event={"ID":"1347bde2-d8a0-48f0-9dc7-826ab61048f5","Type":"ContainerStarted","Data":"79410f74b7c1c9b57a9f80d7e9a0a86f7b04ed7e8939d8d417399bab9eb5b7ce"}
Nov 03 14:32:15.889442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:15.889348    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:16.264363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:16.264331    6481 generic.go:356] "Generic (PLEG): container finished" podID="1347bde2-d8a0-48f0-9dc7-826ab61048f5" containerID="0e6fc7ba8386c436758661c3a9c232937b537debfd4da2416ae19fdd216e0fa1" exitCode=0
Nov 03 14:32:16.264807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:16.264408    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk" event={"ID":"1347bde2-d8a0-48f0-9dc7-826ab61048f5","Type":"ContainerDied","Data":"0e6fc7ba8386c436758661c3a9c232937b537debfd4da2416ae19fdd216e0fa1"}
Nov 03 14:32:17.195722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.195695    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:32:17.459906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.459819    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:17.567262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.565611    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca\") pod \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") "
Nov 03 14:32:17.567262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.565708    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc\" (UniqueName: \"kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc\") pod \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") "
Nov 03 14:32:17.567262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.565780    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca\") pod \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\" (UID: \"1347bde2-d8a0-48f0-9dc7-826ab61048f5\") "
Nov 03 14:32:17.568871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.568371    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca" pod "1347bde2-d8a0-48f0-9dc7-826ab61048f5" (UID: "1347bde2-d8a0-48f0-9dc7-826ab61048f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:17.574273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.570986    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca" pod "1347bde2-d8a0-48f0-9dc7-826ab61048f5" (UID: "1347bde2-d8a0-48f0-9dc7-826ab61048f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:17.574273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.571019    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc" pod "1347bde2-d8a0-48f0-9dc7-826ab61048f5" (UID: "1347bde2-d8a0-48f0-9dc7-826ab61048f5"). InnerVolumeSpecName "kube-api-access-bs4jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:17.682252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.674452    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:17.682252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.674492    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4jc\" (UniqueName: \"kubernetes.io/projected/1347bde2-d8a0-48f0-9dc7-826ab61048f5-kube-api-access-bs4jc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:17.682252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:17.674511    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1347bde2-d8a0-48f0-9dc7-826ab61048f5-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:18.110251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:18.110059    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:18.407438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:18.407410    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:18.551099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:18.551059    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk" event={"ID":"1347bde2-d8a0-48f0-9dc7-826ab61048f5","Type":"ContainerDied","Data":"79410f74b7c1c9b57a9f80d7e9a0a86f7b04ed7e8939d8d417399bab9eb5b7ce"}
Nov 03 14:32:18.551726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:18.551705    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79410f74b7c1c9b57a9f80d7e9a0a86f7b04ed7e8939d8d417399bab9eb5b7ce"
Nov 03 14:32:18.551897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:18.551882    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"
Nov 03 14:32:19.489590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:19.489551    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:20.044394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:20.044358    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"]
Nov 03 14:32:20.055743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:20.055714    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27f21ec-r6p6n"]
Nov 03 14:32:20.068359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:20.068332    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8bde37-0b06-46a8-a187-3569ae35dcbc" path="/var/lib/kubelet/pods/ae8bde37-0b06-46a8-a187-3569ae35dcbc/volumes"
Nov 03 14:32:20.651334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:20.651299    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:21.600916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.600882    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"]
Nov 03 14:32:21.601549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.601530    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1347bde2-d8a0-48f0-9dc7-826ab61048f5" containerName="revoke-keys"
Nov 03 14:32:21.601549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.601548    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1347bde2-d8a0-48f0-9dc7-826ab61048f5" containerName="revoke-keys"
Nov 03 14:32:21.601714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.601677    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1347bde2-d8a0-48f0-9dc7-826ab61048f5" containerName="revoke-keys"
Nov 03 14:32:21.605244 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.605214    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.627988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.627956    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"]
Nov 03 14:32:21.633451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.633425    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:21.633672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.633651    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:21.633787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.633668    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:21.633787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.633704    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:21.667167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.667133    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8htt\" (UniqueName: \"kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.667355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.667248    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.667355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.667339    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.769171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.769129    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8htt\" (UniqueName: \"kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.769422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.769222    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.769422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.769297    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.770071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.770052    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.771837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.771814    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.791034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.791008    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8htt\" (UniqueName: \"kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt\") pod \"maas-api-revoke-sub-057d10ad-gk8g9\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:21.919597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:21.919560    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:22.226923 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:22.226888    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353ecd02_3994_42b2_a4ef_7df261c5f32d.slice/crio-ef02a264e4b05f92ed47ef7a0cfcf761644d2235afc80d1f21730fb4b65c003f WatchSource:0}: Error finding container ef02a264e4b05f92ed47ef7a0cfcf761644d2235afc80d1f21730fb4b65c003f: Status 404 returned error can't find the container with id ef02a264e4b05f92ed47ef7a0cfcf761644d2235afc80d1f21730fb4b65c003f
Nov 03 14:32:22.227553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:22.227480    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"]
Nov 03 14:32:22.586473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:22.586441    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9" event={"ID":"353ecd02-3994-42b2-a4ef-7df261c5f32d","Type":"ContainerStarted","Data":"757faa96cbc2952c460623ce24bc1efbb03c856bdb6dc5a859199489803e01fc"}
Nov 03 14:32:22.586629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:22.586495    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9" event={"ID":"353ecd02-3994-42b2-a4ef-7df261c5f32d","Type":"ContainerStarted","Data":"ef02a264e4b05f92ed47ef7a0cfcf761644d2235afc80d1f21730fb4b65c003f"}
Nov 03 14:32:22.630808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:22.630763    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9" podStartSLOduration=1.63074904 podStartE2EDuration="1.63074904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:22.625921322 +0000 UTC m=+1220.675952505" watchObservedRunningTime="2026-11-03 14:32:22.63074904 +0000 UTC m=+1220.680780222"
Nov 03 14:32:22.836362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:22.836288    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:23.596450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:23.596416    6481 generic.go:356] "Generic (PLEG): container finished" podID="353ecd02-3994-42b2-a4ef-7df261c5f32d" containerID="757faa96cbc2952c460623ce24bc1efbb03c856bdb6dc5a859199489803e01fc" exitCode=0
Nov 03 14:32:23.596623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:23.596496    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9" event={"ID":"353ecd02-3994-42b2-a4ef-7df261c5f32d","Type":"ContainerDied","Data":"757faa96cbc2952c460623ce24bc1efbb03c856bdb6dc5a859199489803e01fc"}
Nov 03 14:32:24.773801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.773778    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:24.924464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.924435    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt\" (UniqueName: \"kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt\") pod \"353ecd02-3994-42b2-a4ef-7df261c5f32d\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") "
Nov 03 14:32:24.924633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.924497    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca\") pod \"353ecd02-3994-42b2-a4ef-7df261c5f32d\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") "
Nov 03 14:32:24.924633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.924551    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca\") pod \"353ecd02-3994-42b2-a4ef-7df261c5f32d\" (UID: \"353ecd02-3994-42b2-a4ef-7df261c5f32d\") "
Nov 03 14:32:24.924907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.924881    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca" pod "353ecd02-3994-42b2-a4ef-7df261c5f32d" (UID: "353ecd02-3994-42b2-a4ef-7df261c5f32d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:24.925108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.925087    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:24.926607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.926581    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca" pod "353ecd02-3994-42b2-a4ef-7df261c5f32d" (UID: "353ecd02-3994-42b2-a4ef-7df261c5f32d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:24.926683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:24.926670    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt" pod "353ecd02-3994-42b2-a4ef-7df261c5f32d" (UID: "353ecd02-3994-42b2-a4ef-7df261c5f32d"). InnerVolumeSpecName "kube-api-access-w8htt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:25.019287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:25.019251    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:25.026861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:25.026828    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8htt\" (UniqueName: \"kubernetes.io/projected/353ecd02-3994-42b2-a4ef-7df261c5f32d-kube-api-access-w8htt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:25.026861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:25.026859    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ecd02-3994-42b2-a4ef-7df261c5f32d-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:25.614051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:25.614023    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"
Nov 03 14:32:25.614199 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:25.614060    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9" event={"ID":"353ecd02-3994-42b2-a4ef-7df261c5f32d","Type":"ContainerDied","Data":"ef02a264e4b05f92ed47ef7a0cfcf761644d2235afc80d1f21730fb4b65c003f"}
Nov 03 14:32:25.614199 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:25.614109    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef02a264e4b05f92ed47ef7a0cfcf761644d2235afc80d1f21730fb4b65c003f"
Nov 03 14:32:26.404153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.404122    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"]
Nov 03 14:32:26.404722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.404708    6481 cpu_manager.go:414] "containerMap: removing container" podUID="353ecd02-3994-42b2-a4ef-7df261c5f32d" containerName="revoke-keys"
Nov 03 14:32:26.404766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.404723    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353ecd02-3994-42b2-a4ef-7df261c5f32d" containerName="revoke-keys"
Nov 03 14:32:26.404872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.404854    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="353ecd02-3994-42b2-a4ef-7df261c5f32d" containerName="revoke-keys"
Nov 03 14:32:26.409075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.409056    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.430391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.430368    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"]
Nov 03 14:32:26.437431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.437409    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:26.437618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.437595    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:26.437731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.437622    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:26.437731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.437624    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:26.545776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.545746    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.545922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.545826    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfz66\" (UniqueName: \"kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.546009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.545988    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.647606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.647576    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.647736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.647641    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.647736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.647701    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfz66\" (UniqueName: \"kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.648348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.648322    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.649960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.649934    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.669214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.669187    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfz66\" (UniqueName: \"kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66\") pod \"maas-api-revoke-sub-ba8620d5-p8wll\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:26.723667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:26.723644    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:27.030675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:27.030650    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"]
Nov 03 14:32:27.030950 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:27.030928    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dcdac5_82ab_474d_83f9_10e268852533.slice/crio-ebebf28c8cc68f9167d97269cdf9f0debed6467f47bcb0eda1ca395d842df9e0 WatchSource:0}: Error finding container ebebf28c8cc68f9167d97269cdf9f0debed6467f47bcb0eda1ca395d842df9e0: Status 404 returned error can't find the container with id ebebf28c8cc68f9167d97269cdf9f0debed6467f47bcb0eda1ca395d842df9e0
Nov 03 14:32:27.180410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:27.180379    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:27.629799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:27.629705    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll" event={"ID":"81dcdac5-82ab-474d-83f9-10e268852533","Type":"ContainerStarted","Data":"ed70e71e2569dad4bd94dc559bd2a14b4a2d4bf0523c74b238ebc93e8531f2e3"}
Nov 03 14:32:27.629799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:27.629763    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll" event={"ID":"81dcdac5-82ab-474d-83f9-10e268852533","Type":"ContainerStarted","Data":"ebebf28c8cc68f9167d97269cdf9f0debed6467f47bcb0eda1ca395d842df9e0"}
Nov 03 14:32:27.671136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:27.671086    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll" podStartSLOduration=1.671072634 podStartE2EDuration="1.671072634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:27.669964204 +0000 UTC m=+1225.719995388" watchObservedRunningTime="2026-11-03 14:32:27.671072634 +0000 UTC m=+1225.721103816"
Nov 03 14:32:27.986018 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:32:27.985987    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:32:28.638574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:28.638540    6481 generic.go:356] "Generic (PLEG): container finished" podID="81dcdac5-82ab-474d-83f9-10e268852533" containerID="ed70e71e2569dad4bd94dc559bd2a14b4a2d4bf0523c74b238ebc93e8531f2e3" exitCode=0
Nov 03 14:32:28.638953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:28.638619    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll" event={"ID":"81dcdac5-82ab-474d-83f9-10e268852533","Type":"ContainerDied","Data":"ed70e71e2569dad4bd94dc559bd2a14b4a2d4bf0523c74b238ebc93e8531f2e3"}
Nov 03 14:32:29.188789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.188761    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:29.805624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.805597    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:29.890421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.890382    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca\") pod \"81dcdac5-82ab-474d-83f9-10e268852533\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") "
Nov 03 14:32:29.890593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.890482    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca\") pod \"81dcdac5-82ab-474d-83f9-10e268852533\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") "
Nov 03 14:32:29.890593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.890587    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66\" (UniqueName: \"kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66\") pod \"81dcdac5-82ab-474d-83f9-10e268852533\" (UID: \"81dcdac5-82ab-474d-83f9-10e268852533\") "
Nov 03 14:32:29.890751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.890726    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca" pod "81dcdac5-82ab-474d-83f9-10e268852533" (UID: "81dcdac5-82ab-474d-83f9-10e268852533"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:29.891146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.891118    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:29.892615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.892588    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca" pod "81dcdac5-82ab-474d-83f9-10e268852533" (UID: "81dcdac5-82ab-474d-83f9-10e268852533"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:29.892768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.892751    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66" pod "81dcdac5-82ab-474d-83f9-10e268852533" (UID: "81dcdac5-82ab-474d-83f9-10e268852533"). InnerVolumeSpecName "kube-api-access-dfz66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:29.992507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.992477    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfz66\" (UniqueName: \"kubernetes.io/projected/81dcdac5-82ab-474d-83f9-10e268852533-kube-api-access-dfz66\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:29.992507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:29.992503    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81dcdac5-82ab-474d-83f9-10e268852533-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:30.657341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:30.657306    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"
Nov 03 14:32:30.657520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:30.657343    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll" event={"ID":"81dcdac5-82ab-474d-83f9-10e268852533","Type":"ContainerDied","Data":"ebebf28c8cc68f9167d97269cdf9f0debed6467f47bcb0eda1ca395d842df9e0"}
Nov 03 14:32:30.657520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:30.657396    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebebf28c8cc68f9167d97269cdf9f0debed6467f47bcb0eda1ca395d842df9e0"
Nov 03 14:32:31.242892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:31.242863    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:31.777897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:31.777860    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:32:32.178189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.178156    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:32.180238 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.180200    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"]
Nov 03 14:32:32.181275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.181253    6481 cpu_manager.go:414] "containerMap: removing container" podUID="81dcdac5-82ab-474d-83f9-10e268852533" containerName="revoke-keys"
Nov 03 14:32:32.181321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.181281    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81dcdac5-82ab-474d-83f9-10e268852533" containerName="revoke-keys"
Nov 03 14:32:32.181489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.181477    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="81dcdac5-82ab-474d-83f9-10e268852533" containerName="revoke-keys"
Nov 03 14:32:32.186201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.186180    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.206487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.206461    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"]
Nov 03 14:32:32.213452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.213429    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:32.213639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.213479    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:32.213639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.213499    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:32.217350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.217332    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:32.326221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.326187    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.326645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.326424    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nbbq\" (UniqueName: \"kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.326645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.326477    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.438304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.438149    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nbbq\" (UniqueName: \"kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.438304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.438248    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.438564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.438382    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.439706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.439679    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.450676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.450641    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.460584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.460557    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nbbq\" (UniqueName: \"kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq\") pod \"maas-api-revoke-sub-f2ece408-jc7qf\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.500196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.500164    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:32.857886 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:32.857851    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f4fcb2_2166_44d0_9fc5_5c02fc4f39bb.slice/crio-c4c11f32b994fd779689b5c1a2102965ab22d58363e6abf8d8e8ac714ea3bba5 WatchSource:0}: Error finding container c4c11f32b994fd779689b5c1a2102965ab22d58363e6abf8d8e8ac714ea3bba5: Status 404 returned error can't find the container with id c4c11f32b994fd779689b5c1a2102965ab22d58363e6abf8d8e8ac714ea3bba5
Nov 03 14:32:32.858138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:32.858116    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"]
Nov 03 14:32:33.682762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:33.682731    6481 generic.go:356] "Generic (PLEG): container finished" podID="b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" containerID="c6f8b81808c1108697b0929017028c9c808198e69729d4778ffd9fac708fc9ce" exitCode=0
Nov 03 14:32:33.683105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:33.682807    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf" event={"ID":"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb","Type":"ContainerDied","Data":"c6f8b81808c1108697b0929017028c9c808198e69729d4778ffd9fac708fc9ce"}
Nov 03 14:32:33.683105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:33.682861    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf" event={"ID":"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb","Type":"ContainerStarted","Data":"c4c11f32b994fd779689b5c1a2102965ab22d58363e6abf8d8e8ac714ea3bba5"}
Nov 03 14:32:33.965012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:33.964946    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:34.041181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.041150    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"]
Nov 03 14:32:34.046861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.046835    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26b20c92-ldc5r"]
Nov 03 14:32:34.067843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.067817    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62e95e0-62fb-49af-a549-dfbca654ab20" path="/var/lib/kubelet/pods/e62e95e0-62fb-49af-a549-dfbca654ab20/volumes"
Nov 03 14:32:34.837177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.837153    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:34.896815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.896777    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca\") pod \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") "
Nov 03 14:32:34.896988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.896950    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca\") pod \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") "
Nov 03 14:32:34.897071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.897055    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq\" (UniqueName: \"kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq\") pod \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\" (UID: \"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb\") "
Nov 03 14:32:34.897362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.897335    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca" pod "b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" (UID: "b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:34.897729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.897706    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:34.899148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.899120    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca" pod "b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" (UID: "b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:34.899148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.899136    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq" pod "b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" (UID: "b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb"). InnerVolumeSpecName "kube-api-access-2nbbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:34.998956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.998868    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:34.998956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:34.998911    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nbbq\" (UniqueName: \"kubernetes.io/projected/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb-kube-api-access-2nbbq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:35.699813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.699787    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"
Nov 03 14:32:35.699969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.699827    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf" event={"ID":"b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb","Type":"ContainerDied","Data":"c4c11f32b994fd779689b5c1a2102965ab22d58363e6abf8d8e8ac714ea3bba5"}
Nov 03 14:32:35.699969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.699873    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c11f32b994fd779689b5c1a2102965ab22d58363e6abf8d8e8ac714ea3bba5"
Nov 03 14:32:35.873359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.873324    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"]
Nov 03 14:32:35.874163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.874141    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" containerName="revoke-keys"
Nov 03 14:32:35.874163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.874163    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" containerName="revoke-keys"
Nov 03 14:32:35.874411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.874396    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" containerName="revoke-keys"
Nov 03 14:32:35.877956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.877937    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:35.894998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.894963    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:35.894998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.894987    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:35.898681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.898658    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"]
Nov 03 14:32:35.908486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.908461    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:35.908618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.908569    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:35.910378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.910349    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:35.910540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.910514    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:35.910631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:35.910617    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngn7z\" (UniqueName: \"kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.012099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.012016    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.012300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.012102    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.012300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.012156    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngn7z\" (UniqueName: \"kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.012727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.012705    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.014447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.014420    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.032930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.032909    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngn7z\" (UniqueName: \"kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z\") pod \"maas-api-revoke-keys-e2e-ait-09d577dc-2z79b\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.192277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.192245    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:36.290562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.290533    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:36.527857 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:36.527822    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302550bc_3bcf_4b92_8db9_1f11c1c92b3c.slice/crio-3e925300ab0e316d87d888c5ea860a57a1ddcc33c2d56474319ff32504a0de93 WatchSource:0}: Error finding container 3e925300ab0e316d87d888c5ea860a57a1ddcc33c2d56474319ff32504a0de93: Status 404 returned error can't find the container with id 3e925300ab0e316d87d888c5ea860a57a1ddcc33c2d56474319ff32504a0de93
Nov 03 14:32:36.528084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.528064    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"]
Nov 03 14:32:36.708817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:36.708788    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b" event={"ID":"302550bc-3bcf-4b92-8db9-1f11c1c92b3c","Type":"ContainerStarted","Data":"3e925300ab0e316d87d888c5ea860a57a1ddcc33c2d56474319ff32504a0de93"}
Nov 03 14:32:37.199954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.199886    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:37.718343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.718311    6481 generic.go:356] "Generic (PLEG): container finished" podID="302550bc-3bcf-4b92-8db9-1f11c1c92b3c" containerID="e20772591d9a6c553e868a302065793bf03c562ace1b8b24882db33fecca8288" exitCode=0
Nov 03 14:32:37.718524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.718396    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b" event={"ID":"302550bc-3bcf-4b92-8db9-1f11c1c92b3c","Type":"ContainerDied","Data":"e20772591d9a6c553e868a302065793bf03c562ace1b8b24882db33fecca8288"}
Nov 03 14:32:37.834240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.834203    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"]
Nov 03 14:32:37.838613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.838595    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:37.860048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.860019    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"]
Nov 03 14:32:37.936135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.936103    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:37.936312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.936165    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhg8n\" (UniqueName: \"kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:37.936432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:37.936411    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.037741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.037664    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhg8n\" (UniqueName: \"kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.037899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.037823    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.037899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.037869    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.038540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.038517    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.040111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.040088    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.058001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.057975    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhg8n\" (UniqueName: \"kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n\") pod \"maas-api-revoke-sub-d9e58365-7f6nj\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.152521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.152492    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:38.459732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.459701    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"]
Nov 03 14:32:38.460565 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:38.460538    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea36e045_2271_4ccf_9e5b_074b532e09ce.slice/crio-2ed811db5ae6bb62e12027edc91fe0613845495f104f184d9164b816a5081566 WatchSource:0}: Error finding container 2ed811db5ae6bb62e12027edc91fe0613845495f104f184d9164b816a5081566: Status 404 returned error can't find the container with id 2ed811db5ae6bb62e12027edc91fe0613845495f104f184d9164b816a5081566
Nov 03 14:32:38.491965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.491926    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:38.728022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.727990    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj" event={"ID":"ea36e045-2271-4ccf-9e5b-074b532e09ce","Type":"ContainerStarted","Data":"2ed811db5ae6bb62e12027edc91fe0613845495f104f184d9164b816a5081566"}
Nov 03 14:32:38.853111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.853069    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:38.949122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.949096    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca\") pod \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") "
Nov 03 14:32:38.949301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.949130    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z\" (UniqueName: \"kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z\") pod \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") "
Nov 03 14:32:38.949301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.949196    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca\") pod \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\" (UID: \"302550bc-3bcf-4b92-8db9-1f11c1c92b3c\") "
Nov 03 14:32:38.949630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.949606    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca" pod "302550bc-3bcf-4b92-8db9-1f11c1c92b3c" (UID: "302550bc-3bcf-4b92-8db9-1f11c1c92b3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:38.949766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.949751    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:38.951384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.951352    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z" pod "302550bc-3bcf-4b92-8db9-1f11c1c92b3c" (UID: "302550bc-3bcf-4b92-8db9-1f11c1c92b3c"). InnerVolumeSpecName "kube-api-access-ngn7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:38.951384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:38.951357    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca" pod "302550bc-3bcf-4b92-8db9-1f11c1c92b3c" (UID: "302550bc-3bcf-4b92-8db9-1f11c1c92b3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:39.051554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.051521    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:39.051554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.051549    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngn7z\" (UniqueName: \"kubernetes.io/projected/302550bc-3bcf-4b92-8db9-1f11c1c92b3c-kube-api-access-ngn7z\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:39.737974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.737936    6481 generic.go:356] "Generic (PLEG): container finished" podID="ea36e045-2271-4ccf-9e5b-074b532e09ce" containerID="3ae2a0b6e7e8136aba9799a1e1dba958471b2ac1eb12ee9e8bea8ce76884e091" exitCode=0
Nov 03 14:32:39.738428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.738017    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj" event={"ID":"ea36e045-2271-4ccf-9e5b-074b532e09ce","Type":"ContainerDied","Data":"3ae2a0b6e7e8136aba9799a1e1dba958471b2ac1eb12ee9e8bea8ce76884e091"}
Nov 03 14:32:39.739467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.739450    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"
Nov 03 14:32:39.739599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.739491    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b" event={"ID":"302550bc-3bcf-4b92-8db9-1f11c1c92b3c","Type":"ContainerDied","Data":"3e925300ab0e316d87d888c5ea860a57a1ddcc33c2d56474319ff32504a0de93"}
Nov 03 14:32:39.739599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:39.739539    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e925300ab0e316d87d888c5ea860a57a1ddcc33c2d56474319ff32504a0de93"
Nov 03 14:32:40.487297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.487266    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:40.906181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.906159    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:40.974128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.974099    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca\") pod \"ea36e045-2271-4ccf-9e5b-074b532e09ce\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") "
Nov 03 14:32:40.974330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.974250    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca\") pod \"ea36e045-2271-4ccf-9e5b-074b532e09ce\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") "
Nov 03 14:32:40.974414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.974343    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n\" (UniqueName: \"kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n\") pod \"ea36e045-2271-4ccf-9e5b-074b532e09ce\" (UID: \"ea36e045-2271-4ccf-9e5b-074b532e09ce\") "
Nov 03 14:32:40.974553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.974519    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca" pod "ea36e045-2271-4ccf-9e5b-074b532e09ce" (UID: "ea36e045-2271-4ccf-9e5b-074b532e09ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:40.974920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.974897    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:40.976323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.976299    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca" pod "ea36e045-2271-4ccf-9e5b-074b532e09ce" (UID: "ea36e045-2271-4ccf-9e5b-074b532e09ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:40.976407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:40.976381    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n" pod "ea36e045-2271-4ccf-9e5b-074b532e09ce" (UID: "ea36e045-2271-4ccf-9e5b-074b532e09ce"). InnerVolumeSpecName "kube-api-access-hhg8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:41.078126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:41.078054    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea36e045-2271-4ccf-9e5b-074b532e09ce-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:41.078126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:41.078082    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhg8n\" (UniqueName: \"kubernetes.io/projected/ea36e045-2271-4ccf-9e5b-074b532e09ce-kube-api-access-hhg8n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:41.755974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:41.755946    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"
Nov 03 14:32:41.756154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:41.755988    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj" event={"ID":"ea36e045-2271-4ccf-9e5b-074b532e09ce","Type":"ContainerDied","Data":"2ed811db5ae6bb62e12027edc91fe0613845495f104f184d9164b816a5081566"}
Nov 03 14:32:41.756154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:41.756041    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ed811db5ae6bb62e12027edc91fe0613845495f104f184d9164b816a5081566"
Nov 03 14:32:42.496938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:42.496905    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:43.131559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.131530    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"]
Nov 03 14:32:43.132110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.132097    6481 cpu_manager.go:414] "containerMap: removing container" podUID="302550bc-3bcf-4b92-8db9-1f11c1c92b3c" containerName="revoke-keys"
Nov 03 14:32:43.132156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.132112    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302550bc-3bcf-4b92-8db9-1f11c1c92b3c" containerName="revoke-keys"
Nov 03 14:32:43.132156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.132137    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ea36e045-2271-4ccf-9e5b-074b532e09ce" containerName="revoke-keys"
Nov 03 14:32:43.132156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.132142    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea36e045-2271-4ccf-9e5b-074b532e09ce" containerName="revoke-keys"
Nov 03 14:32:43.132283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.132223    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea36e045-2271-4ccf-9e5b-074b532e09ce" containerName="revoke-keys"
Nov 03 14:32:43.132283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.132250    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="302550bc-3bcf-4b92-8db9-1f11c1c92b3c" containerName="revoke-keys"
Nov 03 14:32:43.136097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.136082    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.159020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.158996    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"]
Nov 03 14:32:43.164476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.164455    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:43.164580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.164514    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:43.164761 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.164738    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:43.164839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.164804    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:43.209165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.209137    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q99b\" (UniqueName: \"kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.209325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.209181    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.209373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.209323    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.310290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.310260    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.310457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.310418    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q99b\" (UniqueName: \"kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.310457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.310448    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.311046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.311027    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.312603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.312586    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.330918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.330892    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q99b\" (UniqueName: \"kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b\") pod \"maas-api-revoke-sub-5deb936d-5zm72\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.449402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.449367    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:43.756349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.755521    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"]
Nov 03 14:32:43.756771 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:43.756741    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746ed0da_7bc4_4378_93ae_5320e2f1f5fb.slice/crio-5124a8fbc50d62ebc254b7d730218cf046963ce7e09fb6babb08411102e2da56 WatchSource:0}: Error finding container 5124a8fbc50d62ebc254b7d730218cf046963ce7e09fb6babb08411102e2da56: Status 404 returned error can't find the container with id 5124a8fbc50d62ebc254b7d730218cf046963ce7e09fb6babb08411102e2da56
Nov 03 14:32:43.774438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:43.774414    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72" event={"ID":"746ed0da-7bc4-4378-93ae-5320e2f1f5fb","Type":"ContainerStarted","Data":"5124a8fbc50d62ebc254b7d730218cf046963ce7e09fb6babb08411102e2da56"}
Nov 03 14:32:44.723421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:44.723384    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:44.785376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:44.785334    6481 generic.go:356] "Generic (PLEG): container finished" podID="746ed0da-7bc4-4378-93ae-5320e2f1f5fb" containerID="d0ce1a5a6e54c419c7399c1641c6e32f4dfd16d4aa106d4ee2a4d649126ac8a9" exitCode=0
Nov 03 14:32:44.785806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:44.785398    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72" event={"ID":"746ed0da-7bc4-4378-93ae-5320e2f1f5fb","Type":"ContainerDied","Data":"d0ce1a5a6e54c419c7399c1641c6e32f4dfd16d4aa106d4ee2a4d649126ac8a9"}
Nov 03 14:32:44.931014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:44.930986    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:45.944833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:45.944806    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:46.051686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.051604    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca\") pod \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") "
Nov 03 14:32:46.051686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.051652    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b\" (UniqueName: \"kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b\") pod \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") "
Nov 03 14:32:46.051686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.051683    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca\") pod \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\" (UID: \"746ed0da-7bc4-4378-93ae-5320e2f1f5fb\") "
Nov 03 14:32:46.052067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.052036    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca" pod "746ed0da-7bc4-4378-93ae-5320e2f1f5fb" (UID: "746ed0da-7bc4-4378-93ae-5320e2f1f5fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:46.052282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.052261    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:46.053872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.053839    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca" pod "746ed0da-7bc4-4378-93ae-5320e2f1f5fb" (UID: "746ed0da-7bc4-4378-93ae-5320e2f1f5fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:46.053970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.053912    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b" pod "746ed0da-7bc4-4378-93ae-5320e2f1f5fb" (UID: "746ed0da-7bc4-4378-93ae-5320e2f1f5fb"). InnerVolumeSpecName "kube-api-access-7q99b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:46.142343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.142313    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"]
Nov 03 14:32:46.142994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.142975    6481 cpu_manager.go:414] "containerMap: removing container" podUID="746ed0da-7bc4-4378-93ae-5320e2f1f5fb" containerName="revoke-keys"
Nov 03 14:32:46.143096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.142995    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746ed0da-7bc4-4378-93ae-5320e2f1f5fb" containerName="revoke-keys"
Nov 03 14:32:46.143162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.143143    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="746ed0da-7bc4-4378-93ae-5320e2f1f5fb" containerName="revoke-keys"
Nov 03 14:32:46.146919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.146895    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.153139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.153114    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q99b\" (UniqueName: \"kubernetes.io/projected/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-kube-api-access-7q99b\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:46.153139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.153139    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746ed0da-7bc4-4378-93ae-5320e2f1f5fb-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:46.175318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.175299    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"]
Nov 03 14:32:46.254556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254525    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.254700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254672    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.254776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254701    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.254776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254745    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v65cd\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.254947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254847    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.254947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254885    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.255013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.254957    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.255106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.255086    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.255154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.255138    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356628    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356666    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356696    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v65cd\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356714    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356730    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356755    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356837    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.356926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.356876    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.357197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.357002    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.357197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.357082    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.357197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.357154    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.357376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.357261    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.357413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.357372    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.357449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.357437    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.359071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.359051    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.359495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.359477    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.384341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.384312    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v65cd\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.384757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.384738    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.461407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.461379    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:46.785533 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:46.785500    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a634ab_4e7b_4cac_8d9b_71663047f985.slice/crio-7f01827262a5e9caf7f3af9a941680fca0fe3520359a2fa4a3b96565a763261f WatchSource:0}: Error finding container 7f01827262a5e9caf7f3af9a941680fca0fe3520359a2fa4a3b96565a763261f: Status 404 returned error can't find the container with id 7f01827262a5e9caf7f3af9a941680fca0fe3520359a2fa4a3b96565a763261f
Nov 03 14:32:46.785945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.785923    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"]
Nov 03 14:32:46.788034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.788003    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:32:46.788108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.788068    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:32:46.788108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.788098    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:32:46.800867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.800844    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:46.810565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.810543    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"
Nov 03 14:32:46.810672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.810578    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72" event={"ID":"746ed0da-7bc4-4378-93ae-5320e2f1f5fb","Type":"ContainerDied","Data":"5124a8fbc50d62ebc254b7d730218cf046963ce7e09fb6babb08411102e2da56"}
Nov 03 14:32:46.810672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.810629    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5124a8fbc50d62ebc254b7d730218cf046963ce7e09fb6babb08411102e2da56"
Nov 03 14:32:46.812099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:46.812074    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" event={"ID":"35a634ab-4e7b-4cac-8d9b-71663047f985","Type":"ContainerStarted","Data":"7f01827262a5e9caf7f3af9a941680fca0fe3520359a2fa4a3b96565a763261f"}
Nov 03 14:32:47.009818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:47.009782    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:47.319565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:47.319532    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:47.727434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:47.727390    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:47.876267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:47.862119    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" event={"ID":"35a634ab-4e7b-4cac-8d9b-71663047f985","Type":"ContainerStarted","Data":"bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c"}
Nov 03 14:32:47.944583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:47.944502    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podStartSLOduration=1.9444815640000002 podStartE2EDuration="1.944481564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:47.941926882 +0000 UTC m=+1245.991958064" watchObservedRunningTime="2026-11-03 14:32:47.944481564 +0000 UTC m=+1245.994512751"
Nov 03 14:32:48.048664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:48.048571    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"]
Nov 03 14:32:48.052136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:48.052108    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57e85451-77mq4"]
Nov 03 14:32:48.079362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:48.079260    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17eebfe6-45f6-4ca2-9e56-c2043bb54979" path="/var/lib/kubelet/pods/17eebfe6-45f6-4ca2-9e56-c2043bb54979/volumes"
Nov 03 14:32:48.461893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:48.461863    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:48.463091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:48.463063    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 14:32:48.463246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:48.463113    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 14:32:49.059588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:49.059550    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:32:49.461947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:49.461807    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 14:32:49.461947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:49.461857    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 14:32:49.641250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:49.640513    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:50.462181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:50.462144    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 14:32:50.462762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:50.462730    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 14:32:51.154094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:51.154060    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:51.364906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:51.364871    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:51.462705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:51.462673    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 14:32:51.463324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:51.463283    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 14:32:51.577222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:51.577190    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:52.387496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.387468    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:52.462643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.462604    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 14:32:52.462830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.462656    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 14:32:52.470742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.470717    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"]
Nov 03 14:32:52.483835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.483810    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-09d577dc-2z79b"]
Nov 03 14:32:52.618254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.617463    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:52.653723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.653625    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"]
Nov 03 14:32:52.660660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.660632    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.661988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.661948    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"]
Nov 03 14:32:52.678704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.677450    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:32:52.678704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.677691    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:32:52.678704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.677771    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:32:52.678704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.677835    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:32:52.725685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.725646    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfp8h\" (UniqueName: \"kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.726043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.726018    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.726356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.726333    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.827716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.827661    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.828403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.828358    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfp8h\" (UniqueName: \"kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.828532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.828511    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.828600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.828541    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.831607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.831582    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:52.832098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.832081    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.851292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.851269    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfp8h\" (UniqueName: \"kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h\") pod \"maas-api-revoke-sub-fa57fefb-4gcmj\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:52.981780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:52.981749    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:53.271899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.271157    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:53.342800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.342774    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"]
Nov 03 14:32:53.343287 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:53.343262    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980497e0_01b8_4fb0_8810_adf0f8eabe38.slice/crio-9896723e793b3b5df57e24baf839ce2a6b1c4471aa294f29c5090d6a16c7331e WatchSource:0}: Error finding container 9896723e793b3b5df57e24baf839ce2a6b1c4471aa294f29c5090d6a16c7331e: Status 404 returned error can't find the container with id 9896723e793b3b5df57e24baf839ce2a6b1c4471aa294f29c5090d6a16c7331e
Nov 03 14:32:53.462106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.462074    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body=
Nov 03 14:32:53.462319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.462143    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused"
Nov 03 14:32:53.820049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.820007    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"]
Nov 03 14:32:53.825717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.825693    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:53.845032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.845002    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"]
Nov 03 14:32:53.858713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.858687    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:53.939746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.939708    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj" event={"ID":"980497e0-01b8-4fb0-8810-adf0f8eabe38","Type":"ContainerStarted","Data":"9718c398e11d326c7b499b59069ffb2f12725e29e0caaa2fd1a77d8b50444085"}
Nov 03 14:32:53.939894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.939778    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj" event={"ID":"980497e0-01b8-4fb0-8810-adf0f8eabe38","Type":"ContainerStarted","Data":"9896723e793b3b5df57e24baf839ce2a6b1c4471aa294f29c5090d6a16c7331e"}
Nov 03 14:32:53.982465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:53.982418    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj" podStartSLOduration=1.98239988 podStartE2EDuration="1.98239988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:53.981156191 +0000 UTC m=+1252.031187377" watchObservedRunningTime="2026-11-03 14:32:53.98239988 +0000 UTC m=+1252.032431091"
Nov 03 14:32:54.000595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.000556    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.000751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.000681    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwf9\" (UniqueName: \"kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.000837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.000823    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.069565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.069524    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302550bc-3bcf-4b92-8db9-1f11c1c92b3c" path="/var/lib/kubelet/pods/302550bc-3bcf-4b92-8db9-1f11c1c92b3c/volumes"
Nov 03 14:32:54.102608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.102574    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.102798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.102754    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.102878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.102813    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwf9\" (UniqueName: \"kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.103425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.103399    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.105386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.105367    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.124278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.124238    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwf9\" (UniqueName: \"kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9\") pod \"maas-api-revoke-sub-bb85696a-mrnl5\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.156122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.156086    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:54.358483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.358449    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:54.519778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.519700    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"]
Nov 03 14:32:54.524282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.523945    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:54.524891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.524869    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:54.525832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.525809    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:32:54.528693 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:32:54.528670    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843cfeea_9845_4bfc_97fd_03d0cc71cf3a.slice/crio-f1eff108bed7f04d601c1021ddc76d91e92fb215c5a9d9f69bd7e8e5d0c3eca1 WatchSource:0}: Error finding container f1eff108bed7f04d601c1021ddc76d91e92fb215c5a9d9f69bd7e8e5d0c3eca1: Status 404 returned error can't find the container with id f1eff108bed7f04d601c1021ddc76d91e92fb215c5a9d9f69bd7e8e5d0c3eca1
Nov 03 14:32:54.587772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.587746    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:54.949391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.949351    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5" event={"ID":"843cfeea-9845-4bfc-97fd-03d0cc71cf3a","Type":"ContainerStarted","Data":"930ef76828ca7304bd89b8020e2cf6e0adbcb694bce50e5c58306ebddfffda6c"}
Nov 03 14:32:54.949841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.949415    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5" event={"ID":"843cfeea-9845-4bfc-97fd-03d0cc71cf3a","Type":"ContainerStarted","Data":"f1eff108bed7f04d601c1021ddc76d91e92fb215c5a9d9f69bd7e8e5d0c3eca1"}
Nov 03 14:32:54.951005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.950982    6481 generic.go:356] "Generic (PLEG): container finished" podID="980497e0-01b8-4fb0-8810-adf0f8eabe38" containerID="9718c398e11d326c7b499b59069ffb2f12725e29e0caaa2fd1a77d8b50444085" exitCode=0
Nov 03 14:32:54.951127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.951056    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj" event={"ID":"980497e0-01b8-4fb0-8810-adf0f8eabe38","Type":"ContainerDied","Data":"9718c398e11d326c7b499b59069ffb2f12725e29e0caaa2fd1a77d8b50444085"}
Nov 03 14:32:54.994901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:54.994856    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5" podStartSLOduration=1.994842245 podStartE2EDuration="1.994842245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:54.989873288 +0000 UTC m=+1253.039904471" watchObservedRunningTime="2026-11-03 14:32:54.994842245 +0000 UTC m=+1253.044873428"
Nov 03 14:32:55.052012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:55.051980    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"]
Nov 03 14:32:55.059569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:55.059535    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea5f5e2d-zfft8"]
Nov 03 14:32:55.211149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:55.211119    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:55.961188 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:55.961158    6481 generic.go:356] "Generic (PLEG): container finished" podID="843cfeea-9845-4bfc-97fd-03d0cc71cf3a" containerID="930ef76828ca7304bd89b8020e2cf6e0adbcb694bce50e5c58306ebddfffda6c" exitCode=0
Nov 03 14:32:55.961655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:55.961268    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5" event={"ID":"843cfeea-9845-4bfc-97fd-03d0cc71cf3a","Type":"ContainerDied","Data":"930ef76828ca7304bd89b8020e2cf6e0adbcb694bce50e5c58306ebddfffda6c"}
Nov 03 14:32:56.073968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.073932    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="591e01e5-7a96-4a7d-80ce-a51acafd6f2a" path="/var/lib/kubelet/pods/591e01e5-7a96-4a7d-80ce-a51acafd6f2a/volumes"
Nov 03 14:32:56.143053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.143030    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:56.243016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.242606    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca\") pod \"980497e0-01b8-4fb0-8810-adf0f8eabe38\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") "
Nov 03 14:32:56.243016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.242739    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h\" (UniqueName: \"kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h\") pod \"980497e0-01b8-4fb0-8810-adf0f8eabe38\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") "
Nov 03 14:32:56.243016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.242784    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca\") pod \"980497e0-01b8-4fb0-8810-adf0f8eabe38\" (UID: \"980497e0-01b8-4fb0-8810-adf0f8eabe38\") "
Nov 03 14:32:56.244983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.244943    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca" pod "980497e0-01b8-4fb0-8810-adf0f8eabe38" (UID: "980497e0-01b8-4fb0-8810-adf0f8eabe38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:56.255737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.255702    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h" pod "980497e0-01b8-4fb0-8810-adf0f8eabe38" (UID: "980497e0-01b8-4fb0-8810-adf0f8eabe38"). InnerVolumeSpecName "kube-api-access-pfp8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:56.256835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.256810    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca" pod "980497e0-01b8-4fb0-8810-adf0f8eabe38" (UID: "980497e0-01b8-4fb0-8810-adf0f8eabe38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:56.345822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.345790    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfp8h\" (UniqueName: \"kubernetes.io/projected/980497e0-01b8-4fb0-8810-adf0f8eabe38-kube-api-access-pfp8h\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:56.345822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.345815    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:56.345822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.345825    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/980497e0-01b8-4fb0-8810-adf0f8eabe38-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:56.971162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.971122    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj" event={"ID":"980497e0-01b8-4fb0-8810-adf0f8eabe38","Type":"ContainerDied","Data":"9896723e793b3b5df57e24baf839ce2a6b1c4471aa294f29c5090d6a16c7331e"}
Nov 03 14:32:56.971645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.971209    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9896723e793b3b5df57e24baf839ce2a6b1c4471aa294f29c5090d6a16c7331e"
Nov 03 14:32:56.971645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:56.971139    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"
Nov 03 14:32:57.156599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.156569    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:57.264760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.264480    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca\") pod \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") "
Nov 03 14:32:57.264760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.264581    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9\" (UniqueName: \"kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9\") pod \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") "
Nov 03 14:32:57.267865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.266697    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca\") pod \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\" (UID: \"843cfeea-9845-4bfc-97fd-03d0cc71cf3a\") "
Nov 03 14:32:57.270711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.270673    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca" pod "843cfeea-9845-4bfc-97fd-03d0cc71cf3a" (UID: "843cfeea-9845-4bfc-97fd-03d0cc71cf3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:32:57.272629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.272601    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca" pod "843cfeea-9845-4bfc-97fd-03d0cc71cf3a" (UID: "843cfeea-9845-4bfc-97fd-03d0cc71cf3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:32:57.278214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.278172    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9" pod "843cfeea-9845-4bfc-97fd-03d0cc71cf3a" (UID: "843cfeea-9845-4bfc-97fd-03d0cc71cf3a"). InnerVolumeSpecName "kube-api-access-hzwf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:32:57.313075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.313046    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:57.368677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.368644    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:57.368844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.368693    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:57.368844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.368704    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwf9\" (UniqueName: \"kubernetes.io/projected/843cfeea-9845-4bfc-97fd-03d0cc71cf3a-kube-api-access-hzwf9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:32:57.778992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.778963    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:57.983268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.981527    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5" event={"ID":"843cfeea-9845-4bfc-97fd-03d0cc71cf3a","Type":"ContainerDied","Data":"f1eff108bed7f04d601c1021ddc76d91e92fb215c5a9d9f69bd7e8e5d0c3eca1"}
Nov 03 14:32:57.983268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.981623    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1eff108bed7f04d601c1021ddc76d91e92fb215c5a9d9f69bd7e8e5d0c3eca1"
Nov 03 14:32:57.983268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.981691    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"
Nov 03 14:32:57.985077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:57.985052    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:57.988311 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:32:57.988285    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:32:58.242033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:58.242004    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:59.044090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:59.044051    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"]
Nov 03 14:32:59.055171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:59.055142    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-785623f9-4shjh"]
Nov 03 14:32:59.413569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:59.413533    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:59.616640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:59.616612    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:32:59.826048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:32:59.821253    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:00.068240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:00.068195    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99be379e-2cdc-4f36-8413-75bc5f3e788e" path="/var/lib/kubelet/pods/99be379e-2cdc-4f36-8413-75bc5f3e788e/volumes"
Nov 03 14:33:01.397217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:01.397182    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:03.045366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:03.045335    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"]
Nov 03 14:33:03.056505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:03.056475    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7eab127-d8vmb"]
Nov 03 14:33:03.599635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:03.599594    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:04.068514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.068483    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea037f26-764f-4aee-818c-698f952bcad9" path="/var/lib/kubelet/pods/ea037f26-764f-4aee-818c-698f952bcad9/volumes"
Nov 03 14:33:04.235379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.235307    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:04.446960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.446905    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:04.726142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.726053    6481 scope.go:122] "RemoveContainer" containerID="bcaaba40678622f9637d54d894198c951cba70f2a1e8a238277db48a12d6216b"
Nov 03 14:33:04.753723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.753689    6481 scope.go:122] "RemoveContainer" containerID="d7cf71d9071a686c290814ff0a6607a1b012c1f4d5e32bf257fbdccf676502f4"
Nov 03 14:33:04.780982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.780603    6481 scope.go:122] "RemoveContainer" containerID="f799a5bcbc6f65cf00f1645d2378ebfd61501ff97b1d6155f859fb03f1a12c7a"
Nov 03 14:33:04.799708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.799670    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"]
Nov 03 14:33:04.800909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.800882    6481 cpu_manager.go:414] "containerMap: removing container" podUID="980497e0-01b8-4fb0-8810-adf0f8eabe38" containerName="revoke-keys"
Nov 03 14:33:04.801083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.801066    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980497e0-01b8-4fb0-8810-adf0f8eabe38" containerName="revoke-keys"
Nov 03 14:33:04.804967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.802704    6481 cpu_manager.go:414] "containerMap: removing container" podUID="843cfeea-9845-4bfc-97fd-03d0cc71cf3a" containerName="revoke-keys"
Nov 03 14:33:04.804967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.802726    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843cfeea-9845-4bfc-97fd-03d0cc71cf3a" containerName="revoke-keys"
Nov 03 14:33:04.804967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.802897    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="843cfeea-9845-4bfc-97fd-03d0cc71cf3a" containerName="revoke-keys"
Nov 03 14:33:04.804967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.802922    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="980497e0-01b8-4fb0-8810-adf0f8eabe38" containerName="revoke-keys"
Nov 03 14:33:04.814393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.814370    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:04.848099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.848044    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:33:04.848599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.848562    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:33:04.871094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.869243    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:33:04.871094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.869331    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:33:04.871094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.870820    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqsp\" (UniqueName: \"kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:04.871094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.870880    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:04.871094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:04.870929    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.014296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.012700    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"]
Nov 03 14:33:05.014560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.014446    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqsp\" (UniqueName: \"kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.014560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.014511    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.014793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.014558    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.036274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.036166    6481 scope.go:122] "RemoveContainer" containerID="6671ab8156e68b9e97b262957a5a1a757aa3987758b56e02c75f625f71debb5a"
Nov 03 14:33:05.128269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.121628    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.130613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.130582    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.247316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.229049    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqsp\" (UniqueName: \"kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp\") pod \"maas-api-revoke-sub-b75881ae-dgtfk\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.342549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.338606    6481 scope.go:122] "RemoveContainer" containerID="4edfce770acf089152140832d22379cacf7dcb932af891c21582a2cf4c8d7d87"
Nov 03 14:33:05.416145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.415898    6481 scope.go:122] "RemoveContainer" containerID="a37b1cd2c4b3630a34be9c180fe9cd09dbea3d6f4ec1f34d9051d09fb0197f13"
Nov 03 14:33:05.437317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.437172    6481 scope.go:122] "RemoveContainer" containerID="73b77c3ffb4c0ff785c2406140e924e447bd4b91bd3f4f4b0842483423774589"
Nov 03 14:33:05.452026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.451780    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:05.799379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:05.799349    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"]
Nov 03 14:33:05.799753 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:05.799728    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c8b489_a9c0_4059_b654_11baadb6d4a8.slice/crio-43e3352a2dc84329f7b7ad8ed9212e68581117b81ad33809f77886277fd65521 WatchSource:0}: Error finding container 43e3352a2dc84329f7b7ad8ed9212e68581117b81ad33809f77886277fd65521: Status 404 returned error can't find the container with id 43e3352a2dc84329f7b7ad8ed9212e68581117b81ad33809f77886277fd65521
Nov 03 14:33:06.252814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:06.252691    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:06.314661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:06.314627    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk" event={"ID":"22c8b489-a9c0-4059-b654-11baadb6d4a8","Type":"ContainerStarted","Data":"c2b7da830662cd22012ffbd3797eb25ec8cf5e9759b874f703eb8304f309e36a"}
Nov 03 14:33:06.314831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:06.314677    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk" event={"ID":"22c8b489-a9c0-4059-b654-11baadb6d4a8","Type":"ContainerStarted","Data":"43e3352a2dc84329f7b7ad8ed9212e68581117b81ad33809f77886277fd65521"}
Nov 03 14:33:06.356639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:06.356587    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk" podStartSLOduration=2.356573685 podStartE2EDuration="2.356573685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:06.354611202 +0000 UTC m=+1264.404642385" watchObservedRunningTime="2026-11-03 14:33:06.356573685 +0000 UTC m=+1264.406604867"
Nov 03 14:33:07.324420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:07.324388    6481 generic.go:356] "Generic (PLEG): container finished" podID="22c8b489-a9c0-4059-b654-11baadb6d4a8" containerID="c2b7da830662cd22012ffbd3797eb25ec8cf5e9759b874f703eb8304f309e36a" exitCode=0
Nov 03 14:33:07.324861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:07.324475    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk" event={"ID":"22c8b489-a9c0-4059-b654-11baadb6d4a8","Type":"ContainerDied","Data":"c2b7da830662cd22012ffbd3797eb25ec8cf5e9759b874f703eb8304f309e36a"}
Nov 03 14:33:07.704102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:07.703312    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:08.129253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.129168    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:08.330274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.330214    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:08.495866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.495841    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:08.589969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.589936    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca\") pod \"22c8b489-a9c0-4059-b654-11baadb6d4a8\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") "
Nov 03 14:33:08.590157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.590005    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp\" (UniqueName: \"kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp\") pod \"22c8b489-a9c0-4059-b654-11baadb6d4a8\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") "
Nov 03 14:33:08.590157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.590058    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca\") pod \"22c8b489-a9c0-4059-b654-11baadb6d4a8\" (UID: \"22c8b489-a9c0-4059-b654-11baadb6d4a8\") "
Nov 03 14:33:08.590500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.590472    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca" pod "22c8b489-a9c0-4059-b654-11baadb6d4a8" (UID: "22c8b489-a9c0-4059-b654-11baadb6d4a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:08.590645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.590630    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:08.592019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.591991    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca" pod "22c8b489-a9c0-4059-b654-11baadb6d4a8" (UID: "22c8b489-a9c0-4059-b654-11baadb6d4a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:33:08.592534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.592514    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp" pod "22c8b489-a9c0-4059-b654-11baadb6d4a8" (UID: "22c8b489-a9c0-4059-b654-11baadb6d4a8"). InnerVolumeSpecName "kube-api-access-fbqsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:08.692206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.692177    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c8b489-a9c0-4059-b654-11baadb6d4a8-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:08.692206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:08.692205    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqsp\" (UniqueName: \"kubernetes.io/projected/22c8b489-a9c0-4059-b654-11baadb6d4a8-kube-api-access-fbqsp\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:09.342383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:09.342352    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"
Nov 03 14:33:09.342383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:09.342380    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk" event={"ID":"22c8b489-a9c0-4059-b654-11baadb6d4a8","Type":"ContainerDied","Data":"43e3352a2dc84329f7b7ad8ed9212e68581117b81ad33809f77886277fd65521"}
Nov 03 14:33:09.342822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:09.342422    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43e3352a2dc84329f7b7ad8ed9212e68581117b81ad33809f77886277fd65521"
Nov 03 14:33:10.278125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:10.278096    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:12.249211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:12.249173    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:13.458315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:13.458218    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:13.855480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:13.855404    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:14.007681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:14.007646    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:33:14.323839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:14.323803    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:15.906421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:15.906387    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"]
Nov 03 14:33:16.042072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.042036    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"]
Nov 03 14:33:16.046723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.046691    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c676596-j28lc"]
Nov 03 14:33:16.071834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.071804    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88585a1c-f8d9-48d8-81fe-68ac03fd5e2f" path="/var/lib/kubelet/pods/88585a1c-f8d9-48d8-81fe-68ac03fd5e2f/volumes"
Nov 03 14:33:16.151042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.150998    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:16.351884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.351833    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:16.571955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.571924    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:16.793567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:16.793517    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:17.109506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.109404    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:17.219995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.219963    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"]
Nov 03 14:33:17.220602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.220587    6481 cpu_manager.go:414] "containerMap: removing container" podUID="22c8b489-a9c0-4059-b654-11baadb6d4a8" containerName="revoke-keys"
Nov 03 14:33:17.220659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.220603    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c8b489-a9c0-4059-b654-11baadb6d4a8" containerName="revoke-keys"
Nov 03 14:33:17.220709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.220700    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c8b489-a9c0-4059-b654-11baadb6d4a8" containerName="revoke-keys"
Nov 03 14:33:17.225383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.225363    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.245947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.245914    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"]
Nov 03 14:33:17.257255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.251894    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:33:17.257255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.251942    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:33:17.257255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.251942    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:33:17.257255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.251971    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:33:17.260848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.260823    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.261043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.260983    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4nl\" (UniqueName: \"kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.261170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.261074    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.322684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.322654    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:17.363099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.363025    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.363099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.363056    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4nl\" (UniqueName: \"kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.363099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.363085    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.363967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.363942    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.365404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.365378    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.384683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.384658    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4nl\" (UniqueName: \"kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl\") pod \"maas-api-revoke-sub-cd49d7ea-7vc8b\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.530959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.530928    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:17.540941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.540907    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:17.829976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.829944    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"]
Nov 03 14:33:17.854405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:17.854377    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"]
Nov 03 14:33:17.857335 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:17.857309    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe30dc5_d54b_40e7_a787_65276b05329a.slice/crio-70e4f766d968f218d28f4dc45c5a999b232c5de7b45a6a97a091f709e57cba28 WatchSource:0}: Error finding container 70e4f766d968f218d28f4dc45c5a999b232c5de7b45a6a97a091f709e57cba28: Status 404 returned error can't find the container with id 70e4f766d968f218d28f4dc45c5a999b232c5de7b45a6a97a091f709e57cba28
Nov 03 14:33:18.439600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:18.439565    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b" event={"ID":"7fe30dc5-d54b-40e7-a787-65276b05329a","Type":"ContainerStarted","Data":"ec6a3c2680bd7b4df49712dbfa3d6f1df2a3a373778322acd3a14b68c3a965ee"}
Nov 03 14:33:18.439993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:18.439627    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b" event={"ID":"7fe30dc5-d54b-40e7-a787-65276b05329a","Type":"ContainerStarted","Data":"70e4f766d968f218d28f4dc45c5a999b232c5de7b45a6a97a091f709e57cba28"}
Nov 03 14:33:18.482323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:18.482274    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b" podStartSLOduration=1.482260034 podStartE2EDuration="1.482260034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:18.480534109 +0000 UTC m=+1276.530565293" watchObservedRunningTime="2026-11-03 14:33:18.482260034 +0000 UTC m=+1276.532291216"
Nov 03 14:33:19.302503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:19.302459    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:19.449746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:19.449708    6481 generic.go:356] "Generic (PLEG): container finished" podID="7fe30dc5-d54b-40e7-a787-65276b05329a" containerID="ec6a3c2680bd7b4df49712dbfa3d6f1df2a3a373778322acd3a14b68c3a965ee" exitCode=0
Nov 03 14:33:19.450284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:19.449785    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b" event={"ID":"7fe30dc5-d54b-40e7-a787-65276b05329a","Type":"ContainerDied","Data":"ec6a3c2680bd7b4df49712dbfa3d6f1df2a3a373778322acd3a14b68c3a965ee"}
Nov 03 14:33:20.625559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.625533    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:20.716335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.716260    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl\" (UniqueName: \"kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl\") pod \"7fe30dc5-d54b-40e7-a787-65276b05329a\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") "
Nov 03 14:33:20.716335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.716320    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca\") pod \"7fe30dc5-d54b-40e7-a787-65276b05329a\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") "
Nov 03 14:33:20.716563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.716541    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca\") pod \"7fe30dc5-d54b-40e7-a787-65276b05329a\" (UID: \"7fe30dc5-d54b-40e7-a787-65276b05329a\") "
Nov 03 14:33:20.716898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.716867    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca" pod "7fe30dc5-d54b-40e7-a787-65276b05329a" (UID: "7fe30dc5-d54b-40e7-a787-65276b05329a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:20.717047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.717020    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:20.718475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.718450    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl" pod "7fe30dc5-d54b-40e7-a787-65276b05329a" (UID: "7fe30dc5-d54b-40e7-a787-65276b05329a"). InnerVolumeSpecName "kube-api-access-mp4nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:20.718561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.718465    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca" pod "7fe30dc5-d54b-40e7-a787-65276b05329a" (UID: "7fe30dc5-d54b-40e7-a787-65276b05329a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:33:20.818288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.818256    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4nl\" (UniqueName: \"kubernetes.io/projected/7fe30dc5-d54b-40e7-a787-65276b05329a-kube-api-access-mp4nl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:20.818288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.818286    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fe30dc5-d54b-40e7-a787-65276b05329a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:20.907470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:20.907432    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq" podUID="3954c1a4-a4a7-48b3-a635-e9ebc415f25e" containerName="payload-processing" containerID="cri-o://8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a" gracePeriod=25
Nov 03 14:33:21.150983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.150959    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:33:21.222874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.222840    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p\" (UniqueName: \"kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p\") pod \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") "
Nov 03 14:33:21.223074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.223053    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume\") pod \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\" (UID: \"3954c1a4-a4a7-48b3-a635-e9ebc415f25e\") "
Nov 03 14:33:21.223574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.223542    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume" pod "3954c1a4-a4a7-48b3-a635-e9ebc415f25e" (UID: "3954c1a4-a4a7-48b3-a635-e9ebc415f25e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:21.223737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.223715    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:21.225829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.225787    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p" pod "3954c1a4-a4a7-48b3-a635-e9ebc415f25e" (UID: "3954c1a4-a4a7-48b3-a635-e9ebc415f25e"). InnerVolumeSpecName "kube-api-access-z9f5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:21.325243 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.325147    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9f5p\" (UniqueName: \"kubernetes.io/projected/3954c1a4-a4a7-48b3-a635-e9ebc415f25e-kube-api-access-z9f5p\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:21.355904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.355865    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:21.465738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.465709    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"
Nov 03 14:33:21.465935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.465704    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b" event={"ID":"7fe30dc5-d54b-40e7-a787-65276b05329a","Type":"ContainerDied","Data":"70e4f766d968f218d28f4dc45c5a999b232c5de7b45a6a97a091f709e57cba28"}
Nov 03 14:33:21.465935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.465873    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e4f766d968f218d28f4dc45c5a999b232c5de7b45a6a97a091f709e57cba28"
Nov 03 14:33:21.467339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.467312    6481 generic.go:356] "Generic (PLEG): container finished" podID="3954c1a4-a4a7-48b3-a635-e9ebc415f25e" containerID="8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a" exitCode=0
Nov 03 14:33:21.467453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.467355    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"
Nov 03 14:33:21.467453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.467403    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq" event={"ID":"3954c1a4-a4a7-48b3-a635-e9ebc415f25e","Type":"ContainerDied","Data":"8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a"}
Nov 03 14:33:21.467542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.467465    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq" event={"ID":"3954c1a4-a4a7-48b3-a635-e9ebc415f25e","Type":"ContainerDied","Data":"8ad32d1a255562a1c9c99232d52213cce786ba0ab39baa7c6000ca8051d6be1e"}
Nov 03 14:33:21.467542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.467494    6481 scope.go:122] "RemoveContainer" containerID="8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a"
Nov 03 14:33:21.481715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.481693    6481 scope.go:122] "RemoveContainer" containerID="8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a"
Nov 03 14:33:21.482493 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:33:21.482467    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a\": container with ID starting with 8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a not found: ID does not exist" containerID="8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a"
Nov 03 14:33:21.482576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.482511    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a"} err="failed to get container status \"8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a\": rpc error: code = NotFound desc = could not find container \"8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a\": container with ID starting with 8b3aa037e8c12a9d7c2c45e4d0d5a4c9413ac7c497a03ab9a9b2b4ad558aa54a not found: ID does not exist"
Nov 03 14:33:21.537123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.537094    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"]
Nov 03 14:33:21.543873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:21.543850    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-4a67e25f-ffcd799c5-zv8hq"]
Nov 03 14:33:22.068027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:22.068000    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3954c1a4-a4a7-48b3-a635-e9ebc415f25e" path="/var/lib/kubelet/pods/3954c1a4-a4a7-48b3-a635-e9ebc415f25e/volumes"
Nov 03 14:33:22.830337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:22.830282    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt" podUID="a18543d3-a77c-435f-acb9-78fee0d7cf2f" containerName="payload-processing" containerID="cri-o://e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5" gracePeriod=25
Nov 03 14:33:23.106688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.106663    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:33:23.148195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.148171    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume\") pod \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") "
Nov 03 14:33:23.148468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.148439    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials\") pod \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") "
Nov 03 14:33:23.148571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.148534    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8\") pod \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") "
Nov 03 14:33:23.148672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.148592    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume\") pod \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\" (UID: \"a18543d3-a77c-435f-acb9-78fee0d7cf2f\") "
Nov 03 14:33:23.148746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.148675    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume" pod "a18543d3-a77c-435f-acb9-78fee0d7cf2f" (UID: "a18543d3-a77c-435f-acb9-78fee0d7cf2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:23.149166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.149141    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:23.149605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.149581    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume" pod "a18543d3-a77c-435f-acb9-78fee0d7cf2f" (UID: "a18543d3-a77c-435f-acb9-78fee0d7cf2f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:23.151298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.151266    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8" pod "a18543d3-a77c-435f-acb9-78fee0d7cf2f" (UID: "a18543d3-a77c-435f-acb9-78fee0d7cf2f"). InnerVolumeSpecName "kube-api-access-7lfq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:23.151808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.151785    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials" pod "a18543d3-a77c-435f-acb9-78fee0d7cf2f" (UID: "a18543d3-a77c-435f-acb9-78fee0d7cf2f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:23.251404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.251371    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:23.251404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.251405    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfq8\" (UniqueName: \"kubernetes.io/projected/a18543d3-a77c-435f-acb9-78fee0d7cf2f-kube-api-access-7lfq8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:23.251580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.251420    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a18543d3-a77c-435f-acb9-78fee0d7cf2f-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:23.397543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.397446    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:23.484192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.484163    6481 generic.go:356] "Generic (PLEG): container finished" podID="a18543d3-a77c-435f-acb9-78fee0d7cf2f" containerID="e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5" exitCode=0
Nov 03 14:33:23.484362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.484220    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt" event={"ID":"a18543d3-a77c-435f-acb9-78fee0d7cf2f","Type":"ContainerDied","Data":"e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5"}
Nov 03 14:33:23.484362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.484297    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt" event={"ID":"a18543d3-a77c-435f-acb9-78fee0d7cf2f","Type":"ContainerDied","Data":"1114a6755ce0c67fd13bad4248ee1e8d0d791bac19c4bbe7c19cb557525532e1"}
Nov 03 14:33:23.484362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.484318    6481 scope.go:122] "RemoveContainer" containerID="e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5"
Nov 03 14:33:23.484362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.484222    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"
Nov 03 14:33:23.494636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.494619    6481 scope.go:122] "RemoveContainer" containerID="e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5"
Nov 03 14:33:23.494901 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:33:23.494883    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5\": container with ID starting with e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5 not found: ID does not exist" containerID="e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5"
Nov 03 14:33:23.494952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.494909    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5"} err="failed to get container status \"e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5\": rpc error: code = NotFound desc = could not find container \"e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5\": container with ID starting with e835acff7a64dc6cacae372bf4e2f8f8630a6a4c8113527dea4a1b6ac67eaab5 not found: ID does not exist"
Nov 03 14:33:23.544929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.544906    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"]
Nov 03 14:33:23.556254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:23.556218    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-5bc57p7mt"]
Nov 03 14:33:24.067906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:24.067875    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18543d3-a77c-435f-acb9-78fee0d7cf2f" path="/var/lib/kubelet/pods/a18543d3-a77c-435f-acb9-78fee0d7cf2f/volumes"
Nov 03 14:33:25.036487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.036461    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:25.466117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.466081    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:25.581148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581118    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"]
Nov 03 14:33:25.581726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581712    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7fe30dc5-d54b-40e7-a787-65276b05329a" containerName="revoke-keys"
Nov 03 14:33:25.581774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581727    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fe30dc5-d54b-40e7-a787-65276b05329a" containerName="revoke-keys"
Nov 03 14:33:25.581774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581752    6481 cpu_manager.go:414] "containerMap: removing container" podUID="3954c1a4-a4a7-48b3-a635-e9ebc415f25e" containerName="payload-processing"
Nov 03 14:33:25.581774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581758    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3954c1a4-a4a7-48b3-a635-e9ebc415f25e" containerName="payload-processing"
Nov 03 14:33:25.581774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581764    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a18543d3-a77c-435f-acb9-78fee0d7cf2f" containerName="payload-processing"
Nov 03 14:33:25.581774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581769    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18543d3-a77c-435f-acb9-78fee0d7cf2f" containerName="payload-processing"
Nov 03 14:33:25.581942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581861    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fe30dc5-d54b-40e7-a787-65276b05329a" containerName="revoke-keys"
Nov 03 14:33:25.581942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581873    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3954c1a4-a4a7-48b3-a635-e9ebc415f25e" containerName="payload-processing"
Nov 03 14:33:25.581942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.581886    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18543d3-a77c-435f-acb9-78fee0d7cf2f" containerName="payload-processing"
Nov 03 14:33:25.587062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.587037    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.603047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.603026    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-6653eb-openshift-default-dockercfg-dsnq8\""
Nov 03 14:33:25.617955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.617926    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"]
Nov 03 14:33:25.689112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689080    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689117    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vstz5\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689156    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689289    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689341    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689389    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689496    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689544    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.689683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.689669    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.790869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.790785    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.790869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.790827    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.790913    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.790938    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791070    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791166    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791192    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vstz5\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791249    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791291    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791402    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791571    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791640    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791748    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.791878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.791850    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.793326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.793307    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.793582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.793563    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.811413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.811386    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.811622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.811605    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vstz5\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5\") pod \"e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") " pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:25.902380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:25.902358    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:26.224181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.224149    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"]
Nov 03 14:33:26.224675 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:26.224600    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d2a5b6b_b8f1_45fc_9973_850cb9979545.slice/crio-cad168a154e6953adaefbda7c7310a24029226753ede887164d49388d018788b WatchSource:0}: Error finding container cad168a154e6953adaefbda7c7310a24029226753ede887164d49388d018788b: Status 404 returned error can't find the container with id cad168a154e6953adaefbda7c7310a24029226753ede887164d49388d018788b
Nov 03 14:33:26.227285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.227255    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:33:26.227390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.227319    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:33:26.227390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.227349    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:33:26.511218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.511126    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" event={"ID":"6d2a5b6b-b8f1-45fc-9973-850cb9979545","Type":"ContainerStarted","Data":"93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6"}
Nov 03 14:33:26.511218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.511188    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" event={"ID":"6d2a5b6b-b8f1-45fc-9973-850cb9979545","Type":"ContainerStarted","Data":"cad168a154e6953adaefbda7c7310a24029226753ede887164d49388d018788b"}
Nov 03 14:33:26.558289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.558238    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podStartSLOduration=1.5582113 podStartE2EDuration="1.5582113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:26.554725977 +0000 UTC m=+1284.604757161" watchObservedRunningTime="2026-11-03 14:33:26.5582113 +0000 UTC m=+1284.608242483"
Nov 03 14:33:26.903452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:26.903380    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:27.046631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:27.046592    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"]
Nov 03 14:33:27.056625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:27.056574    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77405b85-5n8gc"]
Nov 03 14:33:27.227276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:27.226866    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:27.907253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:27.906873    6481 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 14:33:27.907253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:27.906941    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 14:33:27.989747 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:33:27.989703    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:33:28.102254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:28.078838    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3f021a-a57c-40e2-a672-72a60b5e4b82" path="/var/lib/kubelet/pods/ea3f021a-a57c-40e2-a672-72a60b5e4b82/volumes"
Nov 03 14:33:28.102254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:28.102107    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:28.768073 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:28.768037    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:28.923254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:28.920572    6481 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 14:33:28.923254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:28.920648    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 14:33:28.972570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:28.972530    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:29.425284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:29.424908    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:29.911309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:29.904649    6481 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 14:33:29.911309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:29.904721    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 14:33:30.305309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:30.300449    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:30.917253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:30.915588    6481 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 14:33:30.917253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:30.915646    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 14:33:31.333599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.333517    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"]
Nov 03 14:33:31.338307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.338283    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.355131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.355103    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"]
Nov 03 14:33:31.441192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441155    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnk2z\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441203    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441327    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441361    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441384    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441486    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441539    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441632    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.441698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.441687    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543118 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543084    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543145    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543198    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543223    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnk2z\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543286    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543343    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543385    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543419    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543488    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543701    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543781    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.543861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543814    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.544003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.543982    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.544074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.544004    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.545692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.545671    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.545797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.545780    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.563623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.563598    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.563973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.563950    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnk2z\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z\") pod \"e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") " pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.658943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.658918    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:31.925307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.912253    6481 patch_prober.go:29] interesting pod/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body=
Nov 03 14:33:31.925307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:31.912315    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused"
Nov 03 14:33:32.031791 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:32.031751    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5255b506_ba1d_486d_a398_8ea873423e32.slice/crio-0fda3c4652c7b082a01f7346da592f3ac0fe0cb917640f327dd7d994ddc99811 WatchSource:0}: Error finding container 0fda3c4652c7b082a01f7346da592f3ac0fe0cb917640f327dd7d994ddc99811: Status 404 returned error can't find the container with id 0fda3c4652c7b082a01f7346da592f3ac0fe0cb917640f327dd7d994ddc99811
Nov 03 14:33:32.032734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.032705    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"]
Nov 03 14:33:32.034962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.034931    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:33:32.035067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.034998    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:33:32.035067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.035030    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:33:32.293329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.293195    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:32.658049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.658012    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" event={"ID":"5255b506-ba1d-486d-a398-8ea873423e32","Type":"ContainerStarted","Data":"868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38"}
Nov 03 14:33:32.658204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.658077    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" event={"ID":"5255b506-ba1d-486d-a398-8ea873423e32","Type":"ContainerStarted","Data":"0fda3c4652c7b082a01f7346da592f3ac0fe0cb917640f327dd7d994ddc99811"}
Nov 03 14:33:32.659137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.659117    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:32.941311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.940692    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:32.943675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.942101    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:32.943675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.943348    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:33:32.950995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.950960    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:32.988398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:32.988350    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podStartSLOduration=1.988334985 podStartE2EDuration="1.988334985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:32.707518656 +0000 UTC m=+1290.757549866" watchObservedRunningTime="2026-11-03 14:33:32.988334985 +0000 UTC m=+1291.038366171"
Nov 03 14:33:33.158419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:33.158382    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:33.660523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:33.660487    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 14:33:33.660784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:33.660547    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 14:33:33.836057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:33.836017    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:34.059545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.059453    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:34.082649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.082615    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"]
Nov 03 14:33:34.082881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.082839    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy" containerID="cri-o://ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1" gracePeriod=30
Nov 03 14:33:34.368313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.367756    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:34.659896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.659864    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 14:33:34.660072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.659917    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 14:33:34.789897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:34.789862    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:35.659576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:35.659538    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 14:33:35.659991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:35.659609    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 14:33:36.659534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:36.659498    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 14:33:36.659728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:36.659563    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 14:33:37.019803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:37.019691    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:37.238126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:37.238097    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:37.441275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:37.441248    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:37.660062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:37.660026    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 14:33:37.660484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:37.660085    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 14:33:37.864677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:37.864590    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:33:38.661254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:38.660599    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body=
Nov 03 14:33:38.661254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:38.660659    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused"
Nov 03 14:33:39.390807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.390763    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:39.413632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.413606    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:33:39.507341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.507305    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.507936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.507907    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:33:39.508209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.508128    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.508209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.508167    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.508838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.508191    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.508962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.508945    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.509939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.509351    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.509939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.509767    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:39.510148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.510118    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:33:39.510259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.509853    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.510432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.510416    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.510878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.510766    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:33:39.511076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.511059    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs\") pod \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\" (UID: \"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2\") "
Nov 03 14:33:39.511568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.511345    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:39.511568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.511385    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:33:39.512576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.512475    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.512576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.512500    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.512576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.512514    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.512576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.512526    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.512576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.512538    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.512576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.512551    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.513818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.513449    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:33:39.514036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.514005    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "kube-api-access-c8jsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:39.514157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.514139    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy" pod "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" (UID: "aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:33:39.614481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.614441    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.614481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.614468    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8jsm\" (UniqueName: \"kubernetes.io/projected/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-kube-api-access-c8jsm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.614481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.614478    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:39.663891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.663860    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:39.664416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.664394    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:39.665055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.665036    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:33:39.724724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.724697    6481 generic.go:356] "Generic (PLEG): container finished" podID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerID="ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1" exitCode=0
Nov 03 14:33:39.724900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.724755    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"
Nov 03 14:33:39.724900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.724779    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" event={"ID":"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2","Type":"ContainerDied","Data":"ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1"}
Nov 03 14:33:39.724900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.724829    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47" event={"ID":"aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2","Type":"ContainerDied","Data":"283c45443493edda5002a67030e6561669328e4b542ab20fc0461e5c075c4f68"}
Nov 03 14:33:39.724900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.724849    6481 scope.go:122] "RemoveContainer" containerID="ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1"
Nov 03 14:33:39.738391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.738374    6481 scope.go:122] "RemoveContainer" containerID="ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1"
Nov 03 14:33:39.738692 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:33:39.738672    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1\": container with ID starting with ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1 not found: ID does not exist" containerID="ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1"
Nov 03 14:33:39.738745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.738701    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1"} err="failed to get container status \"ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1\": rpc error: code = NotFound desc = could not find container \"ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1\": container with ID starting with ceeaffd629247730668f3d8848518e43fb5fd5aee93792e5d749f078eb84c9a1 not found: ID does not exist"
Nov 03 14:33:39.802330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.802295    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"]
Nov 03 14:33:39.812899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:39.812863    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4a67e25f-openshift-default-7dcdb7754f-kkk47"]
Nov 03 14:33:40.068213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.068107    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" path="/var/lib/kubelet/pods/aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2/volumes"
Nov 03 14:33:40.802047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.802013    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"]
Nov 03 14:33:40.802618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.802605    6481 cpu_manager.go:414] "containerMap: removing container" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy"
Nov 03 14:33:40.802679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.802620    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy"
Nov 03 14:33:40.802753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.802745    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="aae572f6-eb7d-41d3-8b2e-1b6262a5d5e2" containerName="istio-proxy"
Nov 03 14:33:40.807861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.807845    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:40.825089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.825040    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:33:40.829067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.829042    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"]
Nov 03 14:33:40.835331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.834431    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:33:40.835331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.834471    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:33:40.835331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.834532    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:33:40.934101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.934070    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:40.934318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.934298    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:40.934415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:40.934365    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.035866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.035836    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.036026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.036007    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.036095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.036070    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.036711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.036691    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.038202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.038184    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.057442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.057385    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j\") pod \"maas-api-revoke-sub-13e953b1-795jf\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.121936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.121913    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:41.429010 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:41.428971    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1852b2b0_30e9_4d6e_981b_bb6371406529.slice/crio-630c1e88ab80308629627445c2ac84474d03f180694e2136534bf74e35e0d06f WatchSource:0}: Error finding container 630c1e88ab80308629627445c2ac84474d03f180694e2136534bf74e35e0d06f: Status 404 returned error can't find the container with id 630c1e88ab80308629627445c2ac84474d03f180694e2136534bf74e35e0d06f
Nov 03 14:33:41.429452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.429429    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"]
Nov 03 14:33:41.507127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.507095    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:41.745676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.745648    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf" event={"ID":"1852b2b0-30e9-4d6e-981b-bb6371406529","Type":"ContainerStarted","Data":"9cc398736ea8fb45058f6e4495e5b1771180344ebffc2f0c507ce455ec3f4894"}
Nov 03 14:33:41.745806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.745701    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf" event={"ID":"1852b2b0-30e9-4d6e-981b-bb6371406529","Type":"ContainerStarted","Data":"630c1e88ab80308629627445c2ac84474d03f180694e2136534bf74e35e0d06f"}
Nov 03 14:33:41.785513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:41.785470    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf" podStartSLOduration=1.785456835 podStartE2EDuration="1.785456835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:41.784591466 +0000 UTC m=+1299.834622648" watchObservedRunningTime="2026-11-03 14:33:41.785456835 +0000 UTC m=+1299.835488018"
Nov 03 14:33:42.755176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:42.755145    6481 generic.go:356] "Generic (PLEG): container finished" podID="1852b2b0-30e9-4d6e-981b-bb6371406529" containerID="9cc398736ea8fb45058f6e4495e5b1771180344ebffc2f0c507ce455ec3f4894" exitCode=0
Nov 03 14:33:42.755607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:42.755205    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf" event={"ID":"1852b2b0-30e9-4d6e-981b-bb6371406529","Type":"ContainerDied","Data":"9cc398736ea8fb45058f6e4495e5b1771180344ebffc2f0c507ce455ec3f4894"}
Nov 03 14:33:43.488924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.488892    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:43.542762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.542726    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"]
Nov 03 14:33:43.549958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.549928    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.567541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.567517    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"]
Nov 03 14:33:43.678800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.678767    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.678965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.678858    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8z66\" (UniqueName: \"kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.678965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.678938    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.780291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.780264    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.780742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.780460    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.780742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.780493    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8z66\" (UniqueName: \"kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.781122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.781102    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.782637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.782615    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.801145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.801121    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8z66\" (UniqueName: \"kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66\") pod \"maas-api-revoke-keys-e2e-derive-95af53a1-6phbs\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.863667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.863631    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:43.934797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:43.934766    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:44.085303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.085201    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j\") pod \"1852b2b0-30e9-4d6e-981b-bb6371406529\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") "
Nov 03 14:33:44.085453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.085441    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca\") pod \"1852b2b0-30e9-4d6e-981b-bb6371406529\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") "
Nov 03 14:33:44.085500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.085476    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca\") pod \"1852b2b0-30e9-4d6e-981b-bb6371406529\" (UID: \"1852b2b0-30e9-4d6e-981b-bb6371406529\") "
Nov 03 14:33:44.085899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.085865    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca" pod "1852b2b0-30e9-4d6e-981b-bb6371406529" (UID: "1852b2b0-30e9-4d6e-981b-bb6371406529"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:44.087367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.087344    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j" pod "1852b2b0-30e9-4d6e-981b-bb6371406529" (UID: "1852b2b0-30e9-4d6e-981b-bb6371406529"). InnerVolumeSpecName "kube-api-access-swt9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:44.087510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.087490    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca" pod "1852b2b0-30e9-4d6e-981b-bb6371406529" (UID: "1852b2b0-30e9-4d6e-981b-bb6371406529"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:33:44.176673 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:44.176648    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc082905c_2dc8_4348_8989_3569a8e93db0.slice/crio-b52fc3b81ea2c620b27fd3571536e2966f6c7cd97897c966d55de0dcc9d9cfc9 WatchSource:0}: Error finding container b52fc3b81ea2c620b27fd3571536e2966f6c7cd97897c966d55de0dcc9d9cfc9: Status 404 returned error can't find the container with id b52fc3b81ea2c620b27fd3571536e2966f6c7cd97897c966d55de0dcc9d9cfc9
Nov 03 14:33:44.176913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.176891    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"]
Nov 03 14:33:44.187544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.187521    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:44.187642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.187546    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1852b2b0-30e9-4d6e-981b-bb6371406529-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:44.187642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.187560    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swt9j\" (UniqueName: \"kubernetes.io/projected/1852b2b0-30e9-4d6e-981b-bb6371406529-kube-api-access-swt9j\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:44.774896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.774808    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"
Nov 03 14:33:44.774896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.774849    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf" event={"ID":"1852b2b0-30e9-4d6e-981b-bb6371406529","Type":"ContainerDied","Data":"630c1e88ab80308629627445c2ac84474d03f180694e2136534bf74e35e0d06f"}
Nov 03 14:33:44.775124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.774902    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="630c1e88ab80308629627445c2ac84474d03f180694e2136534bf74e35e0d06f"
Nov 03 14:33:44.776728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.776695    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs" event={"ID":"c082905c-2dc8-4348-8989-3569a8e93db0","Type":"ContainerStarted","Data":"306d630ed6801d84088fede03ad1c5b6db3be787cddb78ee4f8e1bd1c7f9115c"}
Nov 03 14:33:44.776849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.776750    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs" event={"ID":"c082905c-2dc8-4348-8989-3569a8e93db0","Type":"ContainerStarted","Data":"b52fc3b81ea2c620b27fd3571536e2966f6c7cd97897c966d55de0dcc9d9cfc9"}
Nov 03 14:33:44.823336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:44.823276    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs" podStartSLOduration=1.823263882 podStartE2EDuration="1.823263882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:44.821410967 +0000 UTC m=+1302.871442150" watchObservedRunningTime="2026-11-03 14:33:44.823263882 +0000 UTC m=+1302.873295065"
Nov 03 14:33:45.540523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:45.540487    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:45.772010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:45.771983    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:45.785653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:45.785621    6481 generic.go:356] "Generic (PLEG): container finished" podID="c082905c-2dc8-4348-8989-3569a8e93db0" containerID="306d630ed6801d84088fede03ad1c5b6db3be787cddb78ee4f8e1bd1c7f9115c" exitCode=0
Nov 03 14:33:45.785790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:45.785672    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs" event={"ID":"c082905c-2dc8-4348-8989-3569a8e93db0","Type":"ContainerDied","Data":"306d630ed6801d84088fede03ad1c5b6db3be787cddb78ee4f8e1bd1c7f9115c"}
Nov 03 14:33:46.057022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:46.056904    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:46.305161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:46.305131    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:46.942467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:46.942442    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:47.029544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.029507    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca\") pod \"c082905c-2dc8-4348-8989-3569a8e93db0\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") "
Nov 03 14:33:47.029788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.029742    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66\" (UniqueName: \"kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66\") pod \"c082905c-2dc8-4348-8989-3569a8e93db0\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") "
Nov 03 14:33:47.030003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.029979    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca\") pod \"c082905c-2dc8-4348-8989-3569a8e93db0\" (UID: \"c082905c-2dc8-4348-8989-3569a8e93db0\") "
Nov 03 14:33:47.031028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.031001    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca" pod "c082905c-2dc8-4348-8989-3569a8e93db0" (UID: "c082905c-2dc8-4348-8989-3569a8e93db0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:47.032347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.032318    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66" pod "c082905c-2dc8-4348-8989-3569a8e93db0" (UID: "c082905c-2dc8-4348-8989-3569a8e93db0"). InnerVolumeSpecName "kube-api-access-j8z66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:47.032434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.032362    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca" pod "c082905c-2dc8-4348-8989-3569a8e93db0" (UID: "c082905c-2dc8-4348-8989-3569a8e93db0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:33:47.131715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.131679    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:47.131715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.131709    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c082905c-2dc8-4348-8989-3569a8e93db0-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:47.132093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.131724    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8z66\" (UniqueName: \"kubernetes.io/projected/c082905c-2dc8-4348-8989-3569a8e93db0-kube-api-access-j8z66\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:47.379789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.379705    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:47.825686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.824647    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs" event={"ID":"c082905c-2dc8-4348-8989-3569a8e93db0","Type":"ContainerDied","Data":"b52fc3b81ea2c620b27fd3571536e2966f6c7cd97897c966d55de0dcc9d9cfc9"}
Nov 03 14:33:47.825686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.824712    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b52fc3b81ea2c620b27fd3571536e2966f6c7cd97897c966d55de0dcc9d9cfc9"
Nov 03 14:33:47.825686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:47.824785    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"
Nov 03 14:33:48.150554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:48.150462    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:50.140256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:50.140209    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"]
Nov 03 14:33:50.261972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:50.261939    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:50.490915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:50.490879    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:51.406452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:51.406421    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:51.626619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:51.626575    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:52.586706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:52.586670    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:52.817666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:52.817631    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:53.043657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:53.043615    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:54.077607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.077577    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:54.214426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.214381    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"]
Nov 03 14:33:54.215205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.215182    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1852b2b0-30e9-4d6e-981b-bb6371406529" containerName="revoke-keys"
Nov 03 14:33:54.215205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.215201    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1852b2b0-30e9-4d6e-981b-bb6371406529" containerName="revoke-keys"
Nov 03 14:33:54.215414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.215217    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c082905c-2dc8-4348-8989-3569a8e93db0" containerName="revoke-keys"
Nov 03 14:33:54.215414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.215223    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c082905c-2dc8-4348-8989-3569a8e93db0" containerName="revoke-keys"
Nov 03 14:33:54.215414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.215351    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1852b2b0-30e9-4d6e-981b-bb6371406529" containerName="revoke-keys"
Nov 03 14:33:54.215414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.215364    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c082905c-2dc8-4348-8989-3569a8e93db0" containerName="revoke-keys"
Nov 03 14:33:54.219018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.218996    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.251472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.251441    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"]
Nov 03 14:33:54.256285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.256261    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:33:54.256425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.256358    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:33:54.301726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.301704    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:33:54.301962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.301941    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:33:54.331198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.331130    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:54.346587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.346530    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.350176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.350014    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645fd\" (UniqueName: \"kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.350176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.350118    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.451498 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.451467    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.451716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.451624    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645fd\" (UniqueName: \"kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.451716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.451650    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.452354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.452327    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.453978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.453952    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.476730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.476702    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-645fd\" (UniqueName: \"kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd\") pod \"maas-api-revoke-sub-577947f1-x5dbm\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.540730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.540688    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:54.543578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.543558    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:54.884693 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:33:54.884373    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080bf727_2419_432e_a67c_c0980df7f81a.slice/crio-8c1262d7c14fc6ea8f72cb181812407a3fa50ddbc45f9e8938360793e7b7d2b9 WatchSource:0}: Error finding container 8c1262d7c14fc6ea8f72cb181812407a3fa50ddbc45f9e8938360793e7b7d2b9: Status 404 returned error can't find the container with id 8c1262d7c14fc6ea8f72cb181812407a3fa50ddbc45f9e8938360793e7b7d2b9
Nov 03 14:33:54.884693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.884411    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"]
Nov 03 14:33:54.945442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:54.945407    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm" event={"ID":"080bf727-2419-432e-a67c-c0980df7f81a","Type":"ContainerStarted","Data":"8c1262d7c14fc6ea8f72cb181812407a3fa50ddbc45f9e8938360793e7b7d2b9"}
Nov 03 14:33:55.140718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.140649    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c" podUID="3f1ff6e4-3d98-4d65-a051-3391bc23672a" containerName="payload-pre-processing" containerID="cri-o://304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0" gracePeriod=25
Nov 03 14:33:55.354158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.354121    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:55.404540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.404516    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:33:55.471275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.471244    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume\") pod \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") "
Nov 03 14:33:55.471449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.471412    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5\" (UniqueName: \"kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5\") pod \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\" (UID: \"3f1ff6e4-3d98-4d65-a051-3391bc23672a\") "
Nov 03 14:33:55.471723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.471697    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume" pod "3f1ff6e4-3d98-4d65-a051-3391bc23672a" (UID: "3f1ff6e4-3d98-4d65-a051-3391bc23672a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:55.472076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.471986    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f1ff6e4-3d98-4d65-a051-3391bc23672a-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:55.473813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.473792    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5" pod "3f1ff6e4-3d98-4d65-a051-3391bc23672a" (UID: "3f1ff6e4-3d98-4d65-a051-3391bc23672a"). InnerVolumeSpecName "kube-api-access-mxqr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:55.573308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.573275    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxqr5\" (UniqueName: \"kubernetes.io/projected/3f1ff6e4-3d98-4d65-a051-3391bc23672a-kube-api-access-mxqr5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:55.953730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.953700    6481 generic.go:356] "Generic (PLEG): container finished" podID="3f1ff6e4-3d98-4d65-a051-3391bc23672a" containerID="304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0" exitCode=0
Nov 03 14:33:55.953872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.953747    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"
Nov 03 14:33:55.953872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.953790    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c" event={"ID":"3f1ff6e4-3d98-4d65-a051-3391bc23672a","Type":"ContainerDied","Data":"304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0"}
Nov 03 14:33:55.953872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.953844    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c" event={"ID":"3f1ff6e4-3d98-4d65-a051-3391bc23672a","Type":"ContainerDied","Data":"5af5b628b40fc3d209b0ee8343aca36de898c61f362a6d2499b8035114c55da4"}
Nov 03 14:33:55.954048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.953874    6481 scope.go:122] "RemoveContainer" containerID="304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0"
Nov 03 14:33:55.955542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.955519    6481 generic.go:356] "Generic (PLEG): container finished" podID="080bf727-2419-432e-a67c-c0980df7f81a" containerID="74a2070fb6379d4216fc358878feb63989eb83a4a0d628e6b97089c3e52dc1cb" exitCode=0
Nov 03 14:33:55.955667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.955608    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm" event={"ID":"080bf727-2419-432e-a67c-c0980df7f81a","Type":"ContainerDied","Data":"74a2070fb6379d4216fc358878feb63989eb83a4a0d628e6b97089c3e52dc1cb"}
Nov 03 14:33:55.964775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.964756    6481 scope.go:122] "RemoveContainer" containerID="304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0"
Nov 03 14:33:55.965031 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:33:55.965010    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0\": container with ID starting with 304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0 not found: ID does not exist" containerID="304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0"
Nov 03 14:33:55.965077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:55.965040    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0"} err="failed to get container status \"304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0\": rpc error: code = NotFound desc = could not find container \"304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0\": container with ID starting with 304d53a7e0f4ea84fe3863f65eb6af198c9cd44516dc1f314ec2cb477168f4f0 not found: ID does not exist"
Nov 03 14:33:56.010605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:56.010582    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"]
Nov 03 14:33:56.022420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:56.022398    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6653eb-5d84f9cb59-x2s5c"]
Nov 03 14:33:56.068814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:56.068786    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1ff6e4-3d98-4d65-a051-3391bc23672a" path="/var/lib/kubelet/pods/3f1ff6e4-3d98-4d65-a051-3391bc23672a/volumes"
Nov 03 14:33:57.107154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.107132    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:57.128826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.128801    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:57.192459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.192419    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca\") pod \"080bf727-2419-432e-a67c-c0980df7f81a\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") "
Nov 03 14:33:57.192743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.192721    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca\") pod \"080bf727-2419-432e-a67c-c0980df7f81a\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") "
Nov 03 14:33:57.192831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.192760    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd\" (UniqueName: \"kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd\") pod \"080bf727-2419-432e-a67c-c0980df7f81a\" (UID: \"080bf727-2419-432e-a67c-c0980df7f81a\") "
Nov 03 14:33:57.193126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.193094    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca" pod "080bf727-2419-432e-a67c-c0980df7f81a" (UID: "080bf727-2419-432e-a67c-c0980df7f81a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:33:57.193738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.193713    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:57.195196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.195167    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca" pod "080bf727-2419-432e-a67c-c0980df7f81a" (UID: "080bf727-2419-432e-a67c-c0980df7f81a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:33:57.196206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.195656    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd" pod "080bf727-2419-432e-a67c-c0980df7f81a" (UID: "080bf727-2419-432e-a67c-c0980df7f81a"). InnerVolumeSpecName "kube-api-access-645fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:33:57.314129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.314001    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-645fd\" (UniqueName: \"kubernetes.io/projected/080bf727-2419-432e-a67c-c0980df7f81a-kube-api-access-645fd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:57.314129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.314044    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/080bf727-2419-432e-a67c-c0980df7f81a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:33:57.898193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.898159    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:57.993292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.993256    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm" event={"ID":"080bf727-2419-432e-a67c-c0980df7f81a","Type":"ContainerDied","Data":"8c1262d7c14fc6ea8f72cb181812407a3fa50ddbc45f9e8938360793e7b7d2b9"}
Nov 03 14:33:57.993542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.993367    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c1262d7c14fc6ea8f72cb181812407a3fa50ddbc45f9e8938360793e7b7d2b9"
Nov 03 14:33:57.993542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:57.993437    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"
Nov 03 14:33:58.006450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:58.006414    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"]
Nov 03 14:33:58.020202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:58.020171    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-95af53a1-6phbs"]
Nov 03 14:33:58.021285 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:33:58.020631    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:33:58.070839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:58.070810    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c082905c-2dc8-4348-8989-3569a8e93db0" path="/var/lib/kubelet/pods/c082905c-2dc8-4348-8989-3569a8e93db0/volumes"
Nov 03 14:33:59.234502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:59.234458    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:59.442454 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:59.442411    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:33:59.668567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:33:59.668517    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:00.291493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:00.291445    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:02.087912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:02.087882    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:02.615936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:02.615905    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:03.112955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:03.112914    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:03.422740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:03.422707    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:03.619998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:03.619964    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:04.687328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:04.687297    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:05.218755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.218725    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:05.509463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.509355    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:05.562537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.562501    6481 scope.go:122] "RemoveContainer" containerID="e2e44ebd5d528c6f98a4359f58c59c6ecd9d58ab8f91c3e5c0f84803fc4d2103"
Nov 03 14:34:05.572938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.572914    6481 scope.go:122] "RemoveContainer" containerID="f596ff02e35b234d33d77981fab5b337ab79529102eb8c14b3eb5ed07d4e8114"
Nov 03 14:34:05.586296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.586184    6481 scope.go:122] "RemoveContainer" containerID="5d1447ae74a34ef98e096d9aadb5a4aef07ec33c60cc2319ab5b675c788e821a"
Nov 03 14:34:05.599247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.599208    6481 scope.go:122] "RemoveContainer" containerID="e0135342218f9ce6e271b08828617b2ee0fe8e405910e9c22e73784a2a2a8543"
Nov 03 14:34:05.613912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.613886    6481 scope.go:122] "RemoveContainer" containerID="edf727f200dc5768118de55d59386c310e7abb1c5ad53a9a5161e817a8a1b709"
Nov 03 14:34:05.622563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:05.622533    6481 scope.go:122] "RemoveContainer" containerID="bc8193fa0ca97d8adb2ac30b6cebe0d259634d0343933b8c54d511e7ed8b6787"
Nov 03 14:34:06.488257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:06.488207    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:34:07.689485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:07.689418    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:09.781966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:09.781936    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:10.522920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:10.522887    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:10.736042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:10.736011    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:11.426992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:11.426955    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:11.709043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:11.708955    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"]
Nov 03 14:34:11.709297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:11.709266    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy" containerID="cri-o://93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6" gracePeriod=30
Nov 03 14:34:11.845272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:11.844516    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:12.328824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:12.328791    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:12.540821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:12.540793    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:12.947408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:12.947356    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:13.206275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:13.206179    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:13.422282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:13.422256    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:14.607598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:14.607560    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:16.356157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:16.356121    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:16.579213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:16.579169    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:16.834582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:16.834542    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:16.981105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:16.981045    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:34:17.088431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088402    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.088431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088435    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.088644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088458    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.088644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088489    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.088644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088532    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.088817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088653    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.088817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088708    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:17.088817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088720    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:17.088817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088759    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.089037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088854    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:17.089037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088859    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.089037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.088994    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:17.089037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089001    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket\") pod \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\" (UID: \"6d2a5b6b-b8f1-45fc-9973-850cb9979545\") "
Nov 03 14:34:17.089275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089176    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:17.089779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089751    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.089898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089783    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.089898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089798    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.089898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089811    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.089898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.089823    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.091096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.091040    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:34:17.091211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.091192    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "kube-api-access-vstz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:17.091411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.091387    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:17.091411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.091404    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token" pod "6d2a5b6b-b8f1-45fc-9973-850cb9979545" (UID: "6d2a5b6b-b8f1-45fc-9973-850cb9979545"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:17.174816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.174789    6481 generic.go:356] "Generic (PLEG): container finished" podID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerID="93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6" exitCode=0
Nov 03 14:34:17.174953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.174822    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" event={"ID":"6d2a5b6b-b8f1-45fc-9973-850cb9979545","Type":"ContainerDied","Data":"93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6"}
Nov 03 14:34:17.174953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.174836    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"
Nov 03 14:34:17.174953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.174857    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5" event={"ID":"6d2a5b6b-b8f1-45fc-9973-850cb9979545","Type":"ContainerDied","Data":"cad168a154e6953adaefbda7c7310a24029226753ede887164d49388d018788b"}
Nov 03 14:34:17.174953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.174878    6481 scope.go:122] "RemoveContainer" containerID="93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6"
Nov 03 14:34:17.185997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.185978    6481 scope.go:122] "RemoveContainer" containerID="93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6"
Nov 03 14:34:17.186298 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:17.186279    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6\": container with ID starting with 93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6 not found: ID does not exist" containerID="93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6"
Nov 03 14:34:17.186357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.186307    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6"} err="failed to get container status \"93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6\": rpc error: code = NotFound desc = could not find container \"93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6\": container with ID starting with 93cb55e3ba31529ea0770a108320399d4e914fb186312e3005ee6dd63b0410a6 not found: ID does not exist"
Nov 03 14:34:17.191216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.191196    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.191315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.191218    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.191315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.191245    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d2a5b6b-b8f1-45fc-9973-850cb9979545-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.191315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.191259    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vstz5\" (UniqueName: \"kubernetes.io/projected/6d2a5b6b-b8f1-45fc-9973-850cb9979545-kube-api-access-vstz5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:17.244557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.244499    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"]
Nov 03 14:34:17.250777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:17.250756    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-6653eb-openshift-default-6df94d5785-fk9p5"]
Nov 03 14:34:18.068031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:18.068001    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" path="/var/lib/kubelet/pods/6d2a5b6b-b8f1-45fc-9973-850cb9979545/volumes"
Nov 03 14:34:18.175504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:18.175473    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:18.420254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:18.413025    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:19.285333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:19.285299    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:19.487368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:19.487335    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:19.681237 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:19.681204    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:34:21.341459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.341420    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"]
Nov 03 14:34:21.361189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.361160    6481 cpu_manager.go:414] "containerMap: removing container" podUID="3f1ff6e4-3d98-4d65-a051-3391bc23672a" containerName="payload-pre-processing"
Nov 03 14:34:21.363303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.363278    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1ff6e4-3d98-4d65-a051-3391bc23672a" containerName="payload-pre-processing"
Nov 03 14:34:21.363539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.363524    6481 cpu_manager.go:414] "containerMap: removing container" podUID="080bf727-2419-432e-a67c-c0980df7f81a" containerName="revoke-keys"
Nov 03 14:34:21.363918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.363899    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="080bf727-2419-432e-a67c-c0980df7f81a" containerName="revoke-keys"
Nov 03 14:34:21.364054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.364041    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy"
Nov 03 14:34:21.364147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.364135    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy"
Nov 03 14:34:21.364470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.364451    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1ff6e4-3d98-4d65-a051-3391bc23672a" containerName="payload-pre-processing"
Nov 03 14:34:21.364609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.364597    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="080bf727-2419-432e-a67c-c0980df7f81a" containerName="revoke-keys"
Nov 03 14:34:21.364704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.364690    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d2a5b6b-b8f1-45fc-9973-850cb9979545" containerName="istio-proxy"
Nov 03 14:34:21.388256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.385469    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"]
Nov 03 14:34:21.389257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.388964    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.487995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.487965    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\""
Nov 03 14:34:21.489920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.489735    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.489920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.489820    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.489920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.489856    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.489920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.489892    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.489920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.489922    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.490293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.489949    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.490293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.490022    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b5h5\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.490293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.490158    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.490293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.490187    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.491604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.491581    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"]
Nov 03 14:34:21.491848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.491837    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"]
Nov 03 14:34:21.491966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.491787    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.594662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.594578    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.594894    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.594955    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b5h5\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.594988    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595030    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595110    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595143    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595204    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595249    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595325    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595357    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595385    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5c5\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595440    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595490    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595525    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595549    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595578    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595602    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.596733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.595995    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.598691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.598076    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.598691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.598407    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.598691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.598616    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.599050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.599026    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.599374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.599353    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.633254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.623548    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.633254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.630871    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.637270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.634598    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b5h5\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z57r2\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.697081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697025    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697162    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5c5\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697224    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697320    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697371    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697417    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697466    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697490    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697578    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.697671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.697612    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.698199 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.698127    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.698323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.698195    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.698323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.698310    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.698450 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.698411    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.700312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.700286    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.700767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.700743    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.718174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.718149    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.718303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.718262    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5c5\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-qpbjh\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:21.842561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.842519    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:21.876346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:21.876321    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:22.163198 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:22.163172    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7426741c_5396_40d8_acbb_90dc68aa5bdd.slice/crio-14d6c23855a0628a05f5233efd344ea46f80d4c6b7a4ff055a44fcb7dd4202d1 WatchSource:0}: Error finding container 14d6c23855a0628a05f5233efd344ea46f80d4c6b7a4ff055a44fcb7dd4202d1: Status 404 returned error can't find the container with id 14d6c23855a0628a05f5233efd344ea46f80d4c6b7a4ff055a44fcb7dd4202d1
Nov 03 14:34:22.163358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.163290    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"]
Nov 03 14:34:22.165652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.165624    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:22.165719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.165701    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:22.165763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.165731    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:22.206330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.206299    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"]
Nov 03 14:34:22.208974 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:22.208944    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc792c6d3_b155_4230_8458_964c5944780d.slice/crio-9e3f499bb1880a068a6e996feec4160b0fffd1925c6a38474ade716b8a781b87 WatchSource:0}: Error finding container 9e3f499bb1880a068a6e996feec4160b0fffd1925c6a38474ade716b8a781b87: Status 404 returned error can't find the container with id 9e3f499bb1880a068a6e996feec4160b0fffd1925c6a38474ade716b8a781b87
Nov 03 14:34:22.211686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.211652    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:22.211812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.211730    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:22.211812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.211778    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:22.226579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.226551    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:22.298489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.298454    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" event={"ID":"c792c6d3-b155-4230-8458-964c5944780d","Type":"ContainerStarted","Data":"9e3f499bb1880a068a6e996feec4160b0fffd1925c6a38474ade716b8a781b87"}
Nov 03 14:34:22.301700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.301602    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" event={"ID":"7426741c-5396-40d8-acbb-90dc68aa5bdd","Type":"ContainerStarted","Data":"dd6d903bce174570abe23a5de41e873ba1c9664c64d0a65d2c2e68d130c74a63"}
Nov 03 14:34:22.301700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.301660    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" event={"ID":"7426741c-5396-40d8-acbb-90dc68aa5bdd","Type":"ContainerStarted","Data":"14d6c23855a0628a05f5233efd344ea46f80d4c6b7a4ff055a44fcb7dd4202d1"}
Nov 03 14:34:22.346807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.346756    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podStartSLOduration=1.3467380869999999 podStartE2EDuration="1.346738087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:22.345890943 +0000 UTC m=+1340.395922120" watchObservedRunningTime="2026-11-03 14:34:22.346738087 +0000 UTC m=+1340.396769270"
Nov 03 14:34:22.843127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:22.843094    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:23.357802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.357766    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" event={"ID":"c792c6d3-b155-4230-8458-964c5944780d","Type":"ContainerStarted","Data":"98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273"}
Nov 03 14:34:23.437721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.437649    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podStartSLOduration=2.437624544 podStartE2EDuration="2.437624544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:23.42368289 +0000 UTC m=+1341.473714074" watchObservedRunningTime="2026-11-03 14:34:23.437624544 +0000 UTC m=+1341.487655729"
Nov 03 14:34:23.846261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.845554    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:23.846261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.845620    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:23.876967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.876933    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:23.878266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.878239    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:23.878401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:23.878301    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:24.325250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:24.321749    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:24.843678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:24.843640    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:24.844325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:24.844294    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:24.876763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:24.876731    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:24.876904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:24.876792    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:25.057267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:25.057221    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"]
Nov 03 14:34:25.067651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:25.067622    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42aeeda5-76767"]
Nov 03 14:34:25.849283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:25.845620    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:25.849283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:25.845685    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:25.877195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:25.877165    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:25.877351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:25.877208    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:26.068128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:26.068093    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b83a9494-63b1-430f-b828-a59664bb53b2" path="/var/lib/kubelet/pods/b83a9494-63b1-430f-b828-a59664bb53b2/volumes"
Nov 03 14:34:26.362276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:26.354652    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:26.848254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:26.847649    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:26.848254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:26.847718    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:26.876903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:26.876868    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:26.877310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:26.876920    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:27.843895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:27.843854    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:27.844179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:27.844147    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:27.877280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:27.877249    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:27.877600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:27.877295    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:27.987726 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:27.987695    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:34:28.386210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:28.386022    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:28.843452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:28.843415    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:28.843655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:28.843464    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:28.877523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:28.877485    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:28.877866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:28.877545    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:29.681431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.681395    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"]
Nov 03 14:34:29.687917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.687894    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.709786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.709756    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"]
Nov 03 14:34:29.714615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.714590    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:34:29.714769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.714717    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:34:29.714769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.714765    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:34:29.714899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.714821    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:34:29.798963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.798920    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.799141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.798995    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs79p\" (UniqueName: \"kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.799141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.799072    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.843549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.843510    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:29.843725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.843559    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:29.877509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.877474    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:29.877654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.877520    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:29.900206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.900177    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.900378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.900360    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.900457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.900417    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs79p\" (UniqueName: \"kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.900976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.900954    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.902811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.902783    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:29.923473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:29.923452    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs79p\" (UniqueName: \"kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p\") pod \"maas-api-revoke-sub-5b0262a5-fv2x4\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:30.004976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.004895    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:30.313310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.313281    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"]
Nov 03 14:34:30.313818 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:30.313791    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7769d7_f0a3_49aa_aeef_ab8d49bebad9.slice/crio-46d2bfd31ba57ca54b9ff729f0f7efca3d8ec2a8c34b837c1f1f9737f88a56b5 WatchSource:0}: Error finding container 46d2bfd31ba57ca54b9ff729f0f7efca3d8ec2a8c34b837c1f1f9737f88a56b5: Status 404 returned error can't find the container with id 46d2bfd31ba57ca54b9ff729f0f7efca3d8ec2a8c34b837c1f1f9737f88a56b5
Nov 03 14:34:30.393627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.393595    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:30.446321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.446289    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4" event={"ID":"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9","Type":"ContainerStarted","Data":"46d2bfd31ba57ca54b9ff729f0f7efca3d8ec2a8c34b837c1f1f9737f88a56b5"}
Nov 03 14:34:30.843477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.843441    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Nov 03 14:34:30.843623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.843508    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Nov 03 14:34:30.877481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.877407    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body=
Nov 03 14:34:30.877481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:30.877462    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused"
Nov 03 14:34:31.460932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:31.460832    6481 generic.go:356] "Generic (PLEG): container finished" podID="1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" containerID="a39e5a793196baf861c7469eb2f6be99e3c0eb4fa04485ee26ea511c2bb2d2e9" exitCode=0
Nov 03 14:34:31.461456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:31.460925    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4" event={"ID":"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9","Type":"ContainerDied","Data":"a39e5a793196baf861c7469eb2f6be99e3c0eb4fa04485ee26ea511c2bb2d2e9"}
Nov 03 14:34:32.116838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.116808    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:32.117363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.117334    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:32.118346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.118325    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 14:34:32.173883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.173850    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:32.174159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.174132    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:32.175055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.175035    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 14:34:32.508361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.508220    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:32.652007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.651974    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:32.741201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.741118    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p\" (UniqueName: \"kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p\") pod \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") "
Nov 03 14:34:32.741386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.741272    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca\") pod \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") "
Nov 03 14:34:32.741386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.741319    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca\") pod \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\" (UID: \"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9\") "
Nov 03 14:34:32.741730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.741701    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca" pod "1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" (UID: "1a7769d7-f0a3-49aa-aeef-ab8d49bebad9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:32.741992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.741971    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:32.743354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.743332    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p" pod "1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" (UID: "1a7769d7-f0a3-49aa-aeef-ab8d49bebad9"). InnerVolumeSpecName "kube-api-access-fs79p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:32.743428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.743334    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca" pod "1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" (UID: "1a7769d7-f0a3-49aa-aeef-ab8d49bebad9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:34:32.843743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.843710    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs79p\" (UniqueName: \"kubernetes.io/projected/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-kube-api-access-fs79p\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:32.843743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:32.843739    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:33.487680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:33.487651    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"
Nov 03 14:34:33.487841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:33.487678    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4" event={"ID":"1a7769d7-f0a3-49aa-aeef-ab8d49bebad9","Type":"ContainerDied","Data":"46d2bfd31ba57ca54b9ff729f0f7efca3d8ec2a8c34b837c1f1f9737f88a56b5"}
Nov 03 14:34:33.487841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:33.487734    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d2bfd31ba57ca54b9ff729f0f7efca3d8ec2a8c34b837c1f1f9737f88a56b5"
Nov 03 14:34:34.458163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:34.458127    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:34.668337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:34.665925    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:36.052063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.052030    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"]
Nov 03 14:34:36.075807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.075779    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59591aa0-9zrzl"]
Nov 03 14:34:36.403460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.403385    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"]
Nov 03 14:34:36.404010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.403995    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" containerName="revoke-keys"
Nov 03 14:34:36.404057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.404013    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" containerName="revoke-keys"
Nov 03 14:34:36.404122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.404113    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" containerName="revoke-keys"
Nov 03 14:34:36.408362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.408337    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.449641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.449593    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"]
Nov 03 14:34:36.520789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.520749    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.520955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.520803    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.520955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.520889    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.521061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.521041    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.521106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.521094    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.521156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.521112    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.521274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.521254    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.521310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.521286    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.521364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.521311    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc4xd\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622392    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622425    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622452    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc4xd\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622511    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622539    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622561    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622694    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622756    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622782    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.622887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622825    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.623104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.622968    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.623104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.623001    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.623359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.623335    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.623481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.623422    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.625033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.625010    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.625167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.625150    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.643300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.643273    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc4xd\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.643403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.643283    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-r88l9\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.758810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.758766    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:36.781502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:36.781471    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:37.098531 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.097476    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"]
Nov 03 14:34:37.098956 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:37.098614    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8240a2_be3d_4408_880d_0512f957be94.slice/crio-9a6048c9f14fb1466dd9d4fa9c1a07f5315ae807ac1bb90812dc02c380ac875c WatchSource:0}: Error finding container 9a6048c9f14fb1466dd9d4fa9c1a07f5315ae807ac1bb90812dc02c380ac875c: Status 404 returned error can't find the container with id 9a6048c9f14fb1466dd9d4fa9c1a07f5315ae807ac1bb90812dc02c380ac875c
Nov 03 14:34:37.101056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.101028    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:37.101143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.101092    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:37.101143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.101121    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:34:37.519903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.519870    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" event={"ID":"df8240a2-be3d-4408-880d-0512f957be94","Type":"ContainerStarted","Data":"c6ebd796b15d03c20c182f171f87ae4a9ceb8091fa37ade57dfafd7538ed2e5b"}
Nov 03 14:34:37.520092 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.519926    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" event={"ID":"df8240a2-be3d-4408-880d-0512f957be94","Type":"ContainerStarted","Data":"9a6048c9f14fb1466dd9d4fa9c1a07f5315ae807ac1bb90812dc02c380ac875c"}
Nov 03 14:34:37.588533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.588483    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podStartSLOduration=1.588468972 podStartE2EDuration="1.588468972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:37.586606808 +0000 UTC m=+1355.636637992" watchObservedRunningTime="2026-11-03 14:34:37.588468972 +0000 UTC m=+1355.638500217"
Nov 03 14:34:37.759730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:37.759697    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:38.068012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:38.067981    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30f0686-e3ff-418f-834c-f6cab1435b44" path="/var/lib/kubelet/pods/e30f0686-e3ff-418f-834c-f6cab1435b44/volumes"
Nov 03 14:34:38.761994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:38.761961    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:38.762420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:38.762026    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:38.969417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:38.969378    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:39.759134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:39.759099    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:39.759340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:39.759150    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:40.146671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.146596    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"]
Nov 03 14:34:40.151271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.151245    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.170260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.170223    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"]
Nov 03 14:34:40.177597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.177572    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:34:40.177597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.177582    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:34:40.177754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.177633    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:34:40.177791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.177768    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:34:40.281676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.281644    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.281860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.281837    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.281930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.281887    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zw2\" (UniqueName: \"kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.383489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.383454    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.383651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.383499    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zw2\" (UniqueName: \"kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.383651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.383637    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.384276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.384224    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.385796 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.385774    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.429137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.429106    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zw2\" (UniqueName: \"kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2\") pod \"maas-api-revoke-sub-848a860c-5g6jp\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.466008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.465983    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:40.776312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.775667    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:40.776312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.775739    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:40.893541 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:40.893509    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475cb202_4db4_40ef_8872_8d5a3aa8588a.slice/crio-bee8f34cd6d73dfe49ad89a43893f54acd3dd788d1f9dd5d030a7eb540e10646 WatchSource:0}: Error finding container bee8f34cd6d73dfe49ad89a43893f54acd3dd788d1f9dd5d030a7eb540e10646: Status 404 returned error can't find the container with id bee8f34cd6d73dfe49ad89a43893f54acd3dd788d1f9dd5d030a7eb540e10646
Nov 03 14:34:40.894420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:40.894381    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"]
Nov 03 14:34:41.288015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:41.287972    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:41.557943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:41.557896    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp" event={"ID":"475cb202-4db4-40ef-8872-8d5a3aa8588a","Type":"ContainerStarted","Data":"abd966aa1461c31ed9b19e2244a2886cf70b13f1bde5dcb94501bda7a0a4d5e7"}
Nov 03 14:34:41.558090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:41.557961    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp" event={"ID":"475cb202-4db4-40ef-8872-8d5a3aa8588a","Type":"ContainerStarted","Data":"bee8f34cd6d73dfe49ad89a43893f54acd3dd788d1f9dd5d030a7eb540e10646"}
Nov 03 14:34:41.759173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:41.759083    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:41.759173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:41.759145    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:41.955047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:41.955009    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:42.530470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.530416    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp" podStartSLOduration=2.530402036 podStartE2EDuration="2.530402036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:41.614068092 +0000 UTC m=+1359.664099275" watchObservedRunningTime="2026-11-03 14:34:42.530402036 +0000 UTC m=+1360.580433216"
Nov 03 14:34:42.530884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.530660    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"]
Nov 03 14:34:42.530884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.530835    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy" containerID="cri-o://9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28" gracePeriod=30
Nov 03 14:34:42.542641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.542609    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"]
Nov 03 14:34:42.542848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.542824    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy" containerID="cri-o://38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab" gracePeriod=30
Nov 03 14:34:42.547139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.547113    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"]
Nov 03 14:34:42.547312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.547294    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy" containerID="cri-o://7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a" gracePeriod=30
Nov 03 14:34:42.554776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.554748    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"]
Nov 03 14:34:42.554997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.554975    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy" containerID="cri-o://e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be" gracePeriod=30
Nov 03 14:34:42.570136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.570111    6481 generic.go:356] "Generic (PLEG): container finished" podID="475cb202-4db4-40ef-8872-8d5a3aa8588a" containerID="abd966aa1461c31ed9b19e2244a2886cf70b13f1bde5dcb94501bda7a0a4d5e7" exitCode=0
Nov 03 14:34:42.570283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.570169    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp" event={"ID":"475cb202-4db4-40ef-8872-8d5a3aa8588a","Type":"ContainerDied","Data":"abd966aa1461c31ed9b19e2244a2886cf70b13f1bde5dcb94501bda7a0a4d5e7"}
Nov 03 14:34:42.759292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.759261    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:42.759432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:42.759315    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:43.601214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.601167    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:43.751348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.751323    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:43.759256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.759215    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:43.759366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.759269    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:43.872759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.872683    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca\") pod \"475cb202-4db4-40ef-8872-8d5a3aa8588a\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") "
Nov 03 14:34:43.872955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.872934    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2\" (UniqueName: \"kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2\") pod \"475cb202-4db4-40ef-8872-8d5a3aa8588a\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") "
Nov 03 14:34:43.873082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.873063    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca\") pod \"475cb202-4db4-40ef-8872-8d5a3aa8588a\" (UID: \"475cb202-4db4-40ef-8872-8d5a3aa8588a\") "
Nov 03 14:34:43.873482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.873449    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca" pod "475cb202-4db4-40ef-8872-8d5a3aa8588a" (UID: "475cb202-4db4-40ef-8872-8d5a3aa8588a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:43.873629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.873593    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:43.875410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.875381    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca" pod "475cb202-4db4-40ef-8872-8d5a3aa8588a" (UID: "475cb202-4db4-40ef-8872-8d5a3aa8588a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:34:43.875513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.875422    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2" pod "475cb202-4db4-40ef-8872-8d5a3aa8588a" (UID: "475cb202-4db4-40ef-8872-8d5a3aa8588a"). InnerVolumeSpecName "kube-api-access-25zw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:43.978420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.975466    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475cb202-4db4-40ef-8872-8d5a3aa8588a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:43.978420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:43.978297    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25zw2\" (UniqueName: \"kubernetes.io/projected/475cb202-4db4-40ef-8872-8d5a3aa8588a-kube-api-access-25zw2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:44.018298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.018264    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:44.431861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.431824    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"]
Nov 03 14:34:44.432513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.432493    6481 cpu_manager.go:414] "containerMap: removing container" podUID="475cb202-4db4-40ef-8872-8d5a3aa8588a" containerName="revoke-keys"
Nov 03 14:34:44.432513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.432511    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475cb202-4db4-40ef-8872-8d5a3aa8588a" containerName="revoke-keys"
Nov 03 14:34:44.432688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.432637    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="475cb202-4db4-40ef-8872-8d5a3aa8588a" containerName="revoke-keys"
Nov 03 14:34:44.436456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.436439    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.457913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.457880    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"]
Nov 03 14:34:44.485403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.485371    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.485576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.485517    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7q42\" (UniqueName: \"kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.485785 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.485749    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.586883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.586836    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.587045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.586979    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.587114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.587066    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7q42\" (UniqueName: \"kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.587664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.587634    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.589426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.589379    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.591903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.591875    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp" event={"ID":"475cb202-4db4-40ef-8872-8d5a3aa8588a","Type":"ContainerDied","Data":"bee8f34cd6d73dfe49ad89a43893f54acd3dd788d1f9dd5d030a7eb540e10646"}
Nov 03 14:34:44.592020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.591932    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bee8f34cd6d73dfe49ad89a43893f54acd3dd788d1f9dd5d030a7eb540e10646"
Nov 03 14:34:44.592020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.591994    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"
Nov 03 14:34:44.609135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.609114    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7q42\" (UniqueName: \"kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42\") pod \"maas-api-revoke-sub-251476f1-2dkzx\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.702520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.702446    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:44.749895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.749864    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:44.759570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.759541    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:44.759729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:44.759594    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:45.062909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.062874    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"]
Nov 03 14:34:45.063175 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:45.063149    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1af475e_33e4_4521_81a8_c60c3470ed11.slice/crio-2a427af5a4a7c0ef41e2d324ab5501b6c66ba91f8482a3c8b7c9896712602af5 WatchSource:0}: Error finding container 2a427af5a4a7c0ef41e2d324ab5501b6c66ba91f8482a3c8b7c9896712602af5: Status 404 returned error can't find the container with id 2a427af5a4a7c0ef41e2d324ab5501b6c66ba91f8482a3c8b7c9896712602af5
Nov 03 14:34:45.163733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.163702    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:45.375978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.375677    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:45.606238 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.606140    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx" event={"ID":"c1af475e-33e4-4521-81a8-c60c3470ed11","Type":"ContainerStarted","Data":"6c5e9f5a8a035fa001e3eebe4936b8c28ff439da3180b7962f9ec949ef681b2a"}
Nov 03 14:34:45.606238 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.606196    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx" event={"ID":"c1af475e-33e4-4521-81a8-c60c3470ed11","Type":"ContainerStarted","Data":"2a427af5a4a7c0ef41e2d324ab5501b6c66ba91f8482a3c8b7c9896712602af5"}
Nov 03 14:34:45.649023 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.648955    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx" podStartSLOduration=1.64893544 podStartE2EDuration="1.64893544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:45.646363032 +0000 UTC m=+1363.696394215" watchObservedRunningTime="2026-11-03 14:34:45.64893544 +0000 UTC m=+1363.698966623"
Nov 03 14:34:45.759171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.759136    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body=
Nov 03 14:34:45.759368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.759206    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused"
Nov 03 14:34:45.890974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:45.890854    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:46.617727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:46.617686    6481 generic.go:356] "Generic (PLEG): container finished" podID="c1af475e-33e4-4521-81a8-c60c3470ed11" containerID="6c5e9f5a8a035fa001e3eebe4936b8c28ff439da3180b7962f9ec949ef681b2a" exitCode=0
Nov 03 14:34:46.617931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:46.617763    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx" event={"ID":"c1af475e-33e4-4521-81a8-c60c3470ed11","Type":"ContainerDied","Data":"6c5e9f5a8a035fa001e3eebe4936b8c28ff439da3180b7962f9ec949ef681b2a"}
Nov 03 14:34:47.239594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.239562    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:47.240295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.240269    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:47.241238 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.241211    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 14:34:47.848446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.848424    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:47.912243 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.912200    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:34:47.923557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.923534    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:34:47.939533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.939511    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca\") pod \"c1af475e-33e4-4521-81a8-c60c3470ed11\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") "
Nov 03 14:34:47.939677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.939579    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42\" (UniqueName: \"kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42\") pod \"c1af475e-33e4-4521-81a8-c60c3470ed11\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") "
Nov 03 14:34:47.939677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.939667    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca\") pod \"c1af475e-33e4-4521-81a8-c60c3470ed11\" (UID: \"c1af475e-33e4-4521-81a8-c60c3470ed11\") "
Nov 03 14:34:47.940203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.940182    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca" pod "c1af475e-33e4-4521-81a8-c60c3470ed11" (UID: "c1af475e-33e4-4521-81a8-c60c3470ed11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:47.940420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.940247    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:34:47.941954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.941934    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca" pod "c1af475e-33e4-4521-81a8-c60c3470ed11" (UID: "c1af475e-33e4-4521-81a8-c60c3470ed11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:34:47.942050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.942025    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42" pod "c1af475e-33e4-4521-81a8-c60c3470ed11" (UID: "c1af475e-33e4-4521-81a8-c60c3470ed11"). InnerVolumeSpecName "kube-api-access-d7q42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:47.966799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:47.966768    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:34:48.040831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.040800    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.041030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.040845    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.041030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.040873    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.041030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.040933    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.041030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.040965    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.041030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041003    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041037    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041084    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041120    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041146    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041202    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041255    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041296    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.041342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041327    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.041753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041390    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:48.041753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041396    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.041753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041475    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.041753 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041619    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.041979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041811    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.041979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041849    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.042434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.042204    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:48.042824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.042617    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.043381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043074    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:48.043381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.041533    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.043381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043218    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.043381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043293    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.043381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043335    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043385    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043427    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043467    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043508    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043563    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043610    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.043680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043647    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043690    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043728    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043823    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043870    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data\") pod \"585e5a88-4a71-4a33-9788-eebbe8356b8b\" (UID: \"585e5a88-4a71-4a33-9788-eebbe8356b8b\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043904    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.043944    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy\") pod \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\" (UID: \"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044003    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.044046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044036    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs\") pod \"519203e3-e560-4068-b4da-58a90d6d4564\" (UID: \"519203e3-e560-4068-b4da-58a90d6d4564\") "
Nov 03 14:34:48.044544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044067    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket\") pod \"58a590d8-6a27-465e-942b-b6b116dd9cdd\" (UID: \"58a590d8-6a27-465e-942b-b6b116dd9cdd\") "
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044886    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044910    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7q42\" (UniqueName: \"kubernetes.io/projected/c1af475e-33e4-4521-81a8-c60c3470ed11-kube-api-access-d7q42\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044925    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1af475e-33e4-4521-81a8-c60c3470ed11-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044928    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044941    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a590d8-6a27-465e-942b-b6b116dd9cdd-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044958    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044970    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.044985    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585e5a88-4a71-4a33-9788-eebbe8356b8b-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045000    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045013    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045028    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.045787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045167    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.045787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045201    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.045787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045524    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:48.045954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045908    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.047797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.046077    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.047797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.046135    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "kube-api-access-r6vzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.047797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.046189    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:34:48.047797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.047124    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.047797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.047349    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.047797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.047631    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.048958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.048297    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.048958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.048381    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:34:48.048958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.045427    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.048958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.048863    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.048958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.048913    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.049822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.049797    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.049975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.049954    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.050194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.050168    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:34:48.052768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.051444    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.052768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.051480    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token" pod "58a590d8-6a27-465e-942b-b6b116dd9cdd" (UID: "58a590d8-6a27-465e-942b-b6b116dd9cdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.052768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.051784    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.052768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.052115    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.053112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.053090    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "kube-api-access-zhfqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.053511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.053463    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "kube-api-access-c7hp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.053511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.053465    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:34:48.053670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.053538    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy" pod "519203e3-e560-4068-b4da-58a90d6d4564" (UID: "519203e3-e560-4068-b4da-58a90d6d4564"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.054188 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.053571    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn" pod "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" (UID: "57c50b4c-b00e-4e6b-ac89-869bd3e4fde4"). InnerVolumeSpecName "kube-api-access-jfhzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:48.055128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.055101    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy" pod "585e5a88-4a71-4a33-9788-eebbe8356b8b" (UID: "585e5a88-4a71-4a33-9788-eebbe8356b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:34:48.146131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146102    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146130    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146145    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146211    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfqb\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-kube-api-access-zhfqb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146264    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146285    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146294    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146303    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/519203e3-e560-4068-b4da-58a90d6d4564-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146315    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146327    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146339    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146349    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146357    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfhzn\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-kube-api-access-jfhzn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146366    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/519203e3-e560-4068-b4da-58a90d6d4564-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146375    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vzw\" (UniqueName: \"kubernetes.io/projected/58a590d8-6a27-465e-942b-b6b116dd9cdd-kube-api-access-r6vzw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146383    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7hp5\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-kube-api-access-c7hp5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146392    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146401    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146410    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146418    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585e5a88-4a71-4a33-9788-eebbe8356b8b-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146426    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/519203e3-e560-4068-b4da-58a90d6d4564-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146434    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146441    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146448    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146456    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146464    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146473    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585e5a88-4a71-4a33-9788-eebbe8356b8b-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146481    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a590d8-6a27-465e-942b-b6b116dd9cdd-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.146591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.146489    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/519203e3-e560-4068-b4da-58a90d6d4564-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:48.296119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.296085    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:48.636914 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.636885    6481 generic.go:356] "Generic (PLEG): container finished" podID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerID="e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be" exitCode=0
Nov 03 14:34:48.637035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.636947    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"
Nov 03 14:34:48.637035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.636970    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" event={"ID":"585e5a88-4a71-4a33-9788-eebbe8356b8b","Type":"ContainerDied","Data":"e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be"}
Nov 03 14:34:48.637132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.637048    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv" event={"ID":"585e5a88-4a71-4a33-9788-eebbe8356b8b","Type":"ContainerDied","Data":"791eea1024ae5d8ce2c96cd621ffc5f354f03fc342c6f889ca0fcda5dda7d5dd"}
Nov 03 14:34:48.637132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.637080    6481 scope.go:122] "RemoveContainer" containerID="e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be"
Nov 03 14:34:48.638656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.638621    6481 generic.go:356] "Generic (PLEG): container finished" podID="519203e3-e560-4068-b4da-58a90d6d4564" containerID="7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a" exitCode=0
Nov 03 14:34:48.638755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.638652    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" event={"ID":"519203e3-e560-4068-b4da-58a90d6d4564","Type":"ContainerDied","Data":"7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a"}
Nov 03 14:34:48.638755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.638699    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"
Nov 03 14:34:48.638875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.638698    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6" event={"ID":"519203e3-e560-4068-b4da-58a90d6d4564","Type":"ContainerDied","Data":"6e6ad610efd0fd183ed2aa6cf2b9618cd7c2aa74a519611a36823ce088647d72"}
Nov 03 14:34:48.640410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.640382    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"
Nov 03 14:34:48.640410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.640399    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx" event={"ID":"c1af475e-33e4-4521-81a8-c60c3470ed11","Type":"ContainerDied","Data":"2a427af5a4a7c0ef41e2d324ab5501b6c66ba91f8482a3c8b7c9896712602af5"}
Nov 03 14:34:48.640576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.640442    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a427af5a4a7c0ef41e2d324ab5501b6c66ba91f8482a3c8b7c9896712602af5"
Nov 03 14:34:48.642058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.642032    6481 generic.go:356] "Generic (PLEG): container finished" podID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerID="9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28" exitCode=0
Nov 03 14:34:48.642167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.642073    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" event={"ID":"58a590d8-6a27-465e-942b-b6b116dd9cdd","Type":"ContainerDied","Data":"9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28"}
Nov 03 14:34:48.642167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.642116    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"
Nov 03 14:34:48.642167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.642134    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92" event={"ID":"58a590d8-6a27-465e-942b-b6b116dd9cdd","Type":"ContainerDied","Data":"73092d8d2dfc01c1b9338c96327ce49521fcc638fe639e0c5cb8112c7bf408fc"}
Nov 03 14:34:48.643722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.643698    6481 generic.go:356] "Generic (PLEG): container finished" podID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerID="38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab" exitCode=0
Nov 03 14:34:48.643837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.643751    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"
Nov 03 14:34:48.643837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.643777    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" event={"ID":"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4","Type":"ContainerDied","Data":"38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab"}
Nov 03 14:34:48.643837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.643816    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr" event={"ID":"57c50b4c-b00e-4e6b-ac89-869bd3e4fde4","Type":"ContainerDied","Data":"9471814d00b4f57c0336fe6568711da41af0f26ede56e2e7ea818221746febb1"}
Nov 03 14:34:48.649407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.649388    6481 scope.go:122] "RemoveContainer" containerID="a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"
Nov 03 14:34:48.659036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.659015    6481 scope.go:122] "RemoveContainer" containerID="e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be"
Nov 03 14:34:48.659323 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:48.659303    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be\": container with ID starting with e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be not found: ID does not exist" containerID="e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be"
Nov 03 14:34:48.659397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.659332    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be"} err="failed to get container status \"e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be\": rpc error: code = NotFound desc = could not find container \"e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be\": container with ID starting with e581adcdcd2229fd9ce3cb85c38290826a3f388e5720badb15a78314f59513be not found: ID does not exist"
Nov 03 14:34:48.659397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.659348    6481 scope.go:122] "RemoveContainer" containerID="a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"
Nov 03 14:34:48.659576 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:48.659558    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12\": container with ID starting with a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12 not found: ID does not exist" containerID="a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"
Nov 03 14:34:48.659620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.659583    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12"} err="failed to get container status \"a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12\": rpc error: code = NotFound desc = could not find container \"a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12\": container with ID starting with a588340ff8acb1d20bbd55631fcabaffb22d9ce39f93a5ff0a5ed4769358ae12 not found: ID does not exist"
Nov 03 14:34:48.659620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.659598    6481 scope.go:122] "RemoveContainer" containerID="7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a"
Nov 03 14:34:48.670048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.670030    6481 scope.go:122] "RemoveContainer" containerID="7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a"
Nov 03 14:34:48.670367 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:48.670340    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a\": container with ID starting with 7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a not found: ID does not exist" containerID="7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a"
Nov 03 14:34:48.670466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.670377    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a"} err="failed to get container status \"7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a\": rpc error: code = NotFound desc = could not find container \"7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a\": container with ID starting with 7edae781dcfc29c53c255f3064419ea52922a6c1a8a2c9255b0b394c0ced145a not found: ID does not exist"
Nov 03 14:34:48.670466 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.670398    6481 scope.go:122] "RemoveContainer" containerID="9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28"
Nov 03 14:34:48.679677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.679659    6481 scope.go:122] "RemoveContainer" containerID="9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28"
Nov 03 14:34:48.679926 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:48.679897    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28\": container with ID starting with 9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28 not found: ID does not exist" containerID="9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28"
Nov 03 14:34:48.680036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.679936    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28"} err="failed to get container status \"9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28\": rpc error: code = NotFound desc = could not find container \"9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28\": container with ID starting with 9418aa73267232a99871fa8549c9725c103fd9112093be1f84c909320124fb28 not found: ID does not exist"
Nov 03 14:34:48.680036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.679956    6481 scope.go:122] "RemoveContainer" containerID="38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab"
Nov 03 14:34:48.689034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.689016    6481 scope.go:122] "RemoveContainer" containerID="38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab"
Nov 03 14:34:48.689441 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:48.689416    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab\": container with ID starting with 38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab not found: ID does not exist" containerID="38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab"
Nov 03 14:34:48.689779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.689756    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab"} err="failed to get container status \"38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab\": rpc error: code = NotFound desc = could not find container \"38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab\": container with ID starting with 38a0e8c85e2cdffa5bf6a84ed604953cf0f681dd6f1349d59af7f6ef02424bab not found: ID does not exist"
Nov 03 14:34:48.698700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.698674    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"]
Nov 03 14:34:48.708386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.708366    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-7chmr"]
Nov 03 14:34:48.743772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.743749    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"]
Nov 03 14:34:48.749561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.749542    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-8glj6"]
Nov 03 14:34:48.786488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.786461    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"]
Nov 03 14:34:48.792683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.792659    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-s62tv"]
Nov 03 14:34:48.826388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.826332    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"]
Nov 03 14:34:48.834396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.834365    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-b4b92"]
Nov 03 14:34:48.959364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:48.959324    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:49.181699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:49.181637    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:50.068121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.068090    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519203e3-e560-4068-b4da-58a90d6d4564" path="/var/lib/kubelet/pods/519203e3-e560-4068-b4da-58a90d6d4564/volumes"
Nov 03 14:34:50.068725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.068703    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" path="/var/lib/kubelet/pods/57c50b4c-b00e-4e6b-ac89-869bd3e4fde4/volumes"
Nov 03 14:34:50.069260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.069238    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" path="/var/lib/kubelet/pods/585e5a88-4a71-4a33-9788-eebbe8356b8b/volumes"
Nov 03 14:34:50.070046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.070030    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" path="/var/lib/kubelet/pods/58a590d8-6a27-465e-942b-b6b116dd9cdd/volumes"
Nov 03 14:34:50.747662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.747618    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:50.831540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.831511    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"]
Nov 03 14:34:50.832127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832112    6481 cpu_manager.go:414] "containerMap: removing container" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy"
Nov 03 14:34:50.832180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832129    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy"
Nov 03 14:34:50.832180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832142    6481 cpu_manager.go:414] "containerMap: removing container" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy"
Nov 03 14:34:50.832180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832148    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy"
Nov 03 14:34:50.832180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832164    6481 cpu_manager.go:414] "containerMap: removing container" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy"
Nov 03 14:34:50.832180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832169    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy"
Nov 03 14:34:50.832180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832178    6481 cpu_manager.go:414] "containerMap: removing container" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832182    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832189    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c1af475e-33e4-4521-81a8-c60c3470ed11" containerName="revoke-keys"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832194    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1af475e-33e4-4521-81a8-c60c3470ed11" containerName="revoke-keys"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832218    6481 cpu_manager.go:414] "containerMap: removing container" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832222    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832334    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="519203e3-e560-4068-b4da-58a90d6d4564" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832342    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832351    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c50b4c-b00e-4e6b-ac89-869bd3e4fde4" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832358    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="585e5a88-4a71-4a33-9788-eebbe8356b8b" containerName="istio-proxy"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832364    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1af475e-33e4-4521-81a8-c60c3470ed11" containerName="revoke-keys"
Nov 03 14:34:50.832370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.832373    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a590d8-6a27-465e-942b-b6b116dd9cdd" containerName="istio-proxy"
Nov 03 14:34:50.835957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.835935    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:50.856655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.856631    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"]
Nov 03 14:34:50.862437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.862412    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2e8614\""
Nov 03 14:34:50.862437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.862428    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2e8614-dockercfg-cw69d\""
Nov 03 14:34:50.975472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.975399    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:50.994920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.980321    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"]
Nov 03 14:34:50.994920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:50.985564    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.004039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.003937    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"]
Nov 03 14:34:51.013211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.013169    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.013839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.013814    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4jb\" (UniqueName: \"kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb\") pod \"payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.066278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.065978    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"]
Nov 03 14:34:51.070667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.070636    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0da1c2b9-rh9t8"]
Nov 03 14:34:51.115780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.115747    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.115960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.115812    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4jb\" (UniqueName: \"kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb\") pod \"payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.115960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.115896    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxgz8\" (UniqueName: \"kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8\") pod \"payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.116159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.116140    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.116831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.116813    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.136375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.136343    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4jb\" (UniqueName: \"kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb\") pod \"payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.149783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.149758    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.228309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.228275    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.228549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.228524    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxgz8\" (UniqueName: \"kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8\") pod \"payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.229291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.229267    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.250196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.250168    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxgz8\" (UniqueName: \"kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8\") pod \"payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") " pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.314278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.314181    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:51.410651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.409930    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"]
Nov 03 14:34:51.417076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.417049    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.433937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.433907    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:34:51.434060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.434001    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"]
Nov 03 14:34:51.435551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.435530    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:34:51.437346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.436799    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:34:51.437423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.436986    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:34:51.457742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.457716    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"]
Nov 03 14:34:51.461294 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:51.461209    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1445e3be_373b_4729_831c_97f7e437abe2.slice/crio-adf63ce57709350c79704735a7281ed9cb4de6bf4491b29a57d84126322bd003 WatchSource:0}: Error finding container adf63ce57709350c79704735a7281ed9cb4de6bf4491b29a57d84126322bd003: Status 404 returned error can't find the container with id adf63ce57709350c79704735a7281ed9cb4de6bf4491b29a57d84126322bd003
Nov 03 14:34:51.532694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.532671    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfqnm\" (UniqueName: \"kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.532784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.532707    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.532954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.532931    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.627370 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:51.627337    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfa9272_8448_43a6_a84e_a20a720d1b61.slice/crio-2f1067c84dde6aad9410dc74bdbfb540733447e43c0a7b3850ce06bf34067ee2 WatchSource:0}: Error finding container 2f1067c84dde6aad9410dc74bdbfb540733447e43c0a7b3850ce06bf34067ee2: Status 404 returned error can't find the container with id 2f1067c84dde6aad9410dc74bdbfb540733447e43c0a7b3850ce06bf34067ee2
Nov 03 14:34:51.628048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.628021    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"]
Nov 03 14:34:51.633646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.633625    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.633740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.633677    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfqnm\" (UniqueName: \"kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.633740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.633700    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.634381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.634364    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.636105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.636083    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.655755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.655735    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfqnm\" (UniqueName: \"kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm\") pod \"maas-api-revoke-sub-2615f60c-vzktg\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.671173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.671144    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq" event={"ID":"1cfa9272-8448-43a6-a84e-a20a720d1b61","Type":"ContainerStarted","Data":"2f1067c84dde6aad9410dc74bdbfb540733447e43c0a7b3850ce06bf34067ee2"}
Nov 03 14:34:51.672824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.672801    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn" event={"ID":"1445e3be-373b-4729-831c-97f7e437abe2","Type":"ContainerStarted","Data":"274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa"}
Nov 03 14:34:51.672929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.672849    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn" event={"ID":"1445e3be-373b-4729-831c-97f7e437abe2","Type":"ContainerStarted","Data":"adf63ce57709350c79704735a7281ed9cb4de6bf4491b29a57d84126322bd003"}
Nov 03 14:34:51.672929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.672878    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:34:51.714635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.714590    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn" podStartSLOduration=1.7145757270000002 podStartE2EDuration="1.714575727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:51.713871481 +0000 UTC m=+1369.763902666" watchObservedRunningTime="2026-11-03 14:34:51.714575727 +0000 UTC m=+1369.764606911"
Nov 03 14:34:51.749022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.749001    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:51.991255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:51.982377    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:34:52.062752 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:34:52.062716    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3ffbee_ecd5_4dff_aeb9_32919dd0d70d.slice/crio-b8dec14fe250f50809dcfb0648b1f7708cc774b8087ed4cbd68c1f83882180b2 WatchSource:0}: Error finding container b8dec14fe250f50809dcfb0648b1f7708cc774b8087ed4cbd68c1f83882180b2: Status 404 returned error can't find the container with id b8dec14fe250f50809dcfb0648b1f7708cc774b8087ed4cbd68c1f83882180b2
Nov 03 14:34:52.078899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.078871    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d15f2c-0eaf-446f-bd9a-24cabca7378a" path="/var/lib/kubelet/pods/d7d15f2c-0eaf-446f-bd9a-24cabca7378a/volumes"
Nov 03 14:34:52.079334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.079198    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"]
Nov 03 14:34:52.688795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.688756    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq" event={"ID":"1cfa9272-8448-43a6-a84e-a20a720d1b61","Type":"ContainerStarted","Data":"5efe450a0f342d54252e771bbfcbae98bb72b5611e8bfa7dff519fe23e30391e"}
Nov 03 14:34:52.689967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.689942    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:34:52.694263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.692575    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg" event={"ID":"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d","Type":"ContainerStarted","Data":"2abf7b5fcfec1cd0e76c7890e93d561fe21ebbcc9bc3dc7a9eee4e520069be72"}
Nov 03 14:34:52.694263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.692621    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg" event={"ID":"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d","Type":"ContainerStarted","Data":"b8dec14fe250f50809dcfb0648b1f7708cc774b8087ed4cbd68c1f83882180b2"}
Nov 03 14:34:52.778586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.776811    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq" podStartSLOduration=2.776792544 podStartE2EDuration="2.776792544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:52.751025115 +0000 UTC m=+1370.801056299" watchObservedRunningTime="2026-11-03 14:34:52.776792544 +0000 UTC m=+1370.826823729"
Nov 03 14:34:52.806603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:52.806544    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg" podStartSLOduration=1.8065237920000001 podStartE2EDuration="1.806523792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:52.802653459 +0000 UTC m=+1370.852684642" watchObservedRunningTime="2026-11-03 14:34:52.806523792 +0000 UTC m=+1370.856554979"
Nov 03 14:34:53.534086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:53.534053    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:53.709752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:53.709713    6481 generic.go:356] "Generic (PLEG): container finished" podID="0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" containerID="2abf7b5fcfec1cd0e76c7890e93d561fe21ebbcc9bc3dc7a9eee4e520069be72" exitCode=0
Nov 03 14:34:53.709947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:53.709853    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg" event={"ID":"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d","Type":"ContainerDied","Data":"2abf7b5fcfec1cd0e76c7890e93d561fe21ebbcc9bc3dc7a9eee4e520069be72"}
Nov 03 14:34:54.872830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.872798    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:54.925806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.925776    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca\") pod \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") "
Nov 03 14:34:54.926013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.925996    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca\") pod \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") "
Nov 03 14:34:54.926156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.926026    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm\" (UniqueName: \"kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm\") pod \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\" (UID: \"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d\") "
Nov 03 14:34:54.926241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.926198    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca" pod "0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" (UID: "0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:34:54.926717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.926684    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:54.928213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.928191    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca" pod "0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" (UID: "0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:34:54.928688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:54.928661    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm" pod "0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" (UID: "0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d"). InnerVolumeSpecName "kube-api-access-kfqnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:34:55.028222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.028139    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:55.028222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.028174    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfqnm\" (UniqueName: \"kubernetes.io/projected/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d-kube-api-access-kfqnm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:34:55.725367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.725334    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"
Nov 03 14:34:55.725367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.725355    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg" event={"ID":"0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d","Type":"ContainerDied","Data":"b8dec14fe250f50809dcfb0648b1f7708cc774b8087ed4cbd68c1f83882180b2"}
Nov 03 14:34:55.725605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.725400    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8dec14fe250f50809dcfb0648b1f7708cc774b8087ed4cbd68c1f83882180b2"
Nov 03 14:34:55.760830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.760799    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:55.988119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:55.988039    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:56.266069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:56.264778    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:56.692037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:56.692011    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:56.933453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:56.933424    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:57.165653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:57.165618    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:57.390657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:57.390577    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:57.696251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:57.695435    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:57.994651 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:34:57.994568    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:34:58.102522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:58.102493    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:58.336017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:58.335931    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:58.764084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:58.764057    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:59.512251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.511598    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:34:59.674641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.674606    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"]
Nov 03 14:34:59.675236 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.675211    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" containerName="revoke-keys"
Nov 03 14:34:59.675287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.675242    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" containerName="revoke-keys"
Nov 03 14:34:59.675384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.675374    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" containerName="revoke-keys"
Nov 03 14:34:59.679096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.679078    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.706486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.706459    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-b44368991f6c\""
Nov 03 14:34:59.706606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.706553    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\""
Nov 03 14:34:59.706606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.706570    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\""
Nov 03 14:34:59.706606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.706555    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-vdscv\""
Nov 03 14:34:59.706786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.706651    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\""
Nov 03 14:34:59.706786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.706718    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\""
Nov 03 14:34:59.713422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.713396    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"]
Nov 03 14:34:59.813153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.812124    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrzpt\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.813153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.812220    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.813153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.812359    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.813153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.812413    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.914298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.914260    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.914486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.914319    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.914486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.914466    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrzpt\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.914568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.914504    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.915058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.915032    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.915180 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.915143    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.916797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.916773    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.935067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.935041    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrzpt\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:34:59.992492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:34:59.992466    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:35:00.305643 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:00.305611    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254dded1_622a_4ddd_9bf7_856b49eb5118.slice/crio-bfb657a35cba7970f609cf1ae42f38155ec64a0fa7f6b6e485baf0e7c6de1ceb WatchSource:0}: Error finding container bfb657a35cba7970f609cf1ae42f38155ec64a0fa7f6b6e485baf0e7c6de1ceb: Status 404 returned error can't find the container with id bfb657a35cba7970f609cf1ae42f38155ec64a0fa7f6b6e485baf0e7c6de1ceb
Nov 03 14:35:00.306103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.306080    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"]
Nov 03 14:35:00.478767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.478737    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:00.749273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.749217    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:00.775451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.775420    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" event={"ID":"254dded1-622a-4ddd-9bf7-856b49eb5118","Type":"ContainerStarted","Data":"d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f"}
Nov 03 14:35:00.775643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.775470    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" event={"ID":"254dded1-622a-4ddd-9bf7-856b49eb5118","Type":"ContainerStarted","Data":"bfb657a35cba7970f609cf1ae42f38155ec64a0fa7f6b6e485baf0e7c6de1ceb"}
Nov 03 14:35:00.775710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.775660    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:35:00.819882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.819827    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" podStartSLOduration=1.819812759 podStartE2EDuration="1.819812759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:00.815889618 +0000 UTC m=+1378.865920798" watchObservedRunningTime="2026-11-03 14:35:00.819812759 +0000 UTC m=+1378.869843941"
Nov 03 14:35:00.979253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:00.976743    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:01.365548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.365510    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"]
Nov 03 14:35:01.370530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.370510    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.388648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.388623    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"e2e-cfl-d4022b-provider-cred\""
Nov 03 14:35:01.402688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.402651    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"]
Nov 03 14:35:01.469407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.466558    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9fc\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.469407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.466683    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.469407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.466716    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.469407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.466812    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.525301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.524074    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:01.586171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.573645    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9fc\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.586171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.573846    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.586171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.573901    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.586171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.574051    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.586171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.576054    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.594255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.586875    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.594255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.591084    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.606250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.601648    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9fc\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.703097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.703062    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:01.752365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:01.752332    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:01.980828 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:01.980748    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:01.980828 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:01.980774    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:01.981020 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:01.980834    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials podName:254dded1-622a-4ddd-9bf7-856b49eb5118 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:02.48081771 +0000 UTC m=+1380.530848885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118") : secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:02.009499 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:02.009472    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a196183_2ef2_4f59_8e83_a4eb267d7b26.slice/crio-0eb7fe76c28681e037ecc41f7e4ddb5887871f1945a7b724a4dd9d4be12df957 WatchSource:0}: Error finding container 0eb7fe76c28681e037ecc41f7e4ddb5887871f1945a7b724a4dd9d4be12df957: Status 404 returned error can't find the container with id 0eb7fe76c28681e037ecc41f7e4ddb5887871f1945a7b724a4dd9d4be12df957
Nov 03 14:35:02.009658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.009637    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"]
Nov 03 14:35:02.130311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.130284    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:02.150115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.150085    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"]
Nov 03 14:35:02.155335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.155310    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.173485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.173459    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:35:02.173600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.173551    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:35:02.173600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.173559    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:35:02.173600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.173589    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:35:02.174463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.174402    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"]
Nov 03 14:35:02.288189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.288099    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.288348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.288318    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqwv\" (UniqueName: \"kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.288406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.288368    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.389662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.389620    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.389844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.389823    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqwv\" (UniqueName: \"kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.389904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.389884    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.390393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.390372    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.392337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.392317    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.415305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.415281    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqwv\" (UniqueName: \"kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv\") pod \"maas-api-revoke-sub-3260b640-zwq9p\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.469122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.469101    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:02.491044 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:02.491020    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:02.491044 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:02.491043    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:02.491222 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:02.491109    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials podName:254dded1-622a-4ddd-9bf7-856b49eb5118 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:03.491090444 +0000 UTC m=+1381.541121620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118") : secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:02.695376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.695350    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:35:02.783362 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:02.783332    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5ba7b8_7eb7_43a0_85bb_419cf4c3ec0b.slice/crio-537d1138c36fef20350af1520b67f75f40ff7879871ec49a126f095a7dcacc56 WatchSource:0}: Error finding container 537d1138c36fef20350af1520b67f75f40ff7879871ec49a126f095a7dcacc56: Status 404 returned error can't find the container with id 537d1138c36fef20350af1520b67f75f40ff7879871ec49a126f095a7dcacc56
Nov 03 14:35:02.784127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.784096    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"]
Nov 03 14:35:02.801602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.801573    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb" event={"ID":"9a196183-2ef2-4f59-8e83-a4eb267d7b26","Type":"ContainerStarted","Data":"ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584"}
Nov 03 14:35:02.801723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.801628    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb" event={"ID":"9a196183-2ef2-4f59-8e83-a4eb267d7b26","Type":"ContainerStarted","Data":"0eb7fe76c28681e037ecc41f7e4ddb5887871f1945a7b724a4dd9d4be12df957"}
Nov 03 14:35:02.801723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.801696    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:02.803122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.803100    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p" event={"ID":"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b","Type":"ContainerStarted","Data":"537d1138c36fef20350af1520b67f75f40ff7879871ec49a126f095a7dcacc56"}
Nov 03 14:35:02.846966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.846918    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb" podStartSLOduration=1.846905102 podStartE2EDuration="1.846905102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:02.842859633 +0000 UTC m=+1380.892890816" watchObservedRunningTime="2026-11-03 14:35:02.846905102 +0000 UTC m=+1380.896936284"
Nov 03 14:35:02.974730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:02.974655    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:03.503334 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:03.503302    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:03.503334 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:03.503328    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:03.503542 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:03.503379    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials podName:254dded1-622a-4ddd-9bf7-856b49eb5118 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:05.503364874 +0000 UTC m=+1383.553396035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118") : secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:03.819608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:03.819581    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p" event={"ID":"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b","Type":"ContainerStarted","Data":"41f2d78fff7655197c4da08ba567ba047c78c057296f663d4cba1ca02e7a2e3e"}
Nov 03 14:35:04.719493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:04.719463    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:35:04.829193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:04.829159    6481 generic.go:356] "Generic (PLEG): container finished" podID="6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" containerID="41f2d78fff7655197c4da08ba567ba047c78c057296f663d4cba1ca02e7a2e3e" exitCode=0
Nov 03 14:35:04.829683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:04.829276    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p" event={"ID":"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b","Type":"ContainerDied","Data":"41f2d78fff7655197c4da08ba567ba047c78c057296f663d4cba1ca02e7a2e3e"}
Nov 03 14:35:05.006687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.006660    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:05.023919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.023891    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca\") pod \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") "
Nov 03 14:35:05.024089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.023942    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv\" (UniqueName: \"kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv\") pod \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") "
Nov 03 14:35:05.024089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.024073    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca\") pod \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\" (UID: \"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b\") "
Nov 03 14:35:05.024473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.024445    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca" pod "6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" (UID: "6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:35:05.024680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.024664    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:05.026039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.026010    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca" pod "6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" (UID: "6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:35:05.026178 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.026158    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv" pod "6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" (UID: "6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b"). InnerVolumeSpecName "kube-api-access-6xqwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:35:05.026449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.026413    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:05.042721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.042698    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"]
Nov 03 14:35:05.051194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.051143    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848771fc-mbcsp"]
Nov 03 14:35:05.126287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.126261    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:05.126287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.126285    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqwv\" (UniqueName: \"kubernetes.io/projected/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b-kube-api-access-6xqwv\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:05.532139 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:05.532106    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:05.532139 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:05.532133    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:05.532382 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:05.532184    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials podName:254dded1-622a-4ddd-9bf7-856b49eb5118 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:09.532170372 +0000 UTC m=+1387.582201535 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118") : secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:05.618851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.618821    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"]
Nov 03 14:35:05.619464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.619434    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" containerName="revoke-keys"
Nov 03 14:35:05.619464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.619450    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" containerName="revoke-keys"
Nov 03 14:35:05.619662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.619587    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" containerName="revoke-keys"
Nov 03 14:35:05.623361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.623341    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.639309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.639288    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4a0349\"/\"default-dockercfg-82lrw\""
Nov 03 14:35:05.639437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.639325    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4a0349\"/\"explicit-model-4a0349-kserve-self-signed-certs\""
Nov 03 14:35:05.643585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.643562    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"]
Nov 03 14:35:05.650037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.650014    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4a0349\"/\"openshift-service-ca.crt\""
Nov 03 14:35:05.650037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.650016    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4a0349\"/\"kube-root-ca.crt\""
Nov 03 14:35:05.691814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.691786    6481 scope.go:122] "RemoveContainer" containerID="a538a24ced8e1ae6ce776fec6c3a678095e4f94280116258795fb0a3132436ac"
Nov 03 14:35:05.704741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.704720    6481 scope.go:122] "RemoveContainer" containerID="e5095e820438e9203275f776482c07ae4b5fcfddd654910af4a4289a2592bc7d"
Nov 03 14:35:05.734827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.734799    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.734940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.734850    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.735079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.735001    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.735079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.735029    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.735079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.735062    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.735301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.735134    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk48h\" (UniqueName: \"kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.786682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.786631    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:05.836736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.836706    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.836763    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.836863    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.836892    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.837046    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.837107    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk48h\" (UniqueName: \"kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.837129    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.837169    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.837565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.837290    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.838453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.838433    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"
Nov 03 14:35:05.838575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.838461    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p" event={"ID":"6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b","Type":"ContainerDied","Data":"537d1138c36fef20350af1520b67f75f40ff7879871ec49a126f095a7dcacc56"}
Nov 03 14:35:05.838575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.838505    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="537d1138c36fef20350af1520b67f75f40ff7879871ec49a126f095a7dcacc56"
Nov 03 14:35:05.839399 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.839377    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.839885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.839861    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.857872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.857852    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk48h\" (UniqueName: \"kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h\") pod \"explicit-model-4a0349-kserve-c5c45d85c-vpttv\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") " pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:05.952297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:05.944496    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:06.068887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.068814    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5cd5f80-9d5a-4779-9f71-c874dbf99c51" path="/var/lib/kubelet/pods/d5cd5f80-9d5a-4779-9f71-c874dbf99c51/volumes"
Nov 03 14:35:06.276174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.276142    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"]
Nov 03 14:35:06.849740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.849701    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" event={"ID":"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0","Type":"ContainerStarted","Data":"c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6"}
Nov 03 14:35:06.850129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.849752    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" event={"ID":"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0","Type":"ContainerStarted","Data":"b29a01af3c189bc34dfedaa6c796526e28d00b97f4ef6f5c439910a64b2cebac"}
Nov 03 14:35:06.894539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.894488    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" podStartSLOduration=1.8944748 podStartE2EDuration="1.8944748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:06.89171603 +0000 UTC m=+1384.941747212" watchObservedRunningTime="2026-11-03 14:35:06.8944748 +0000 UTC m=+1384.944505983"
Nov 03 14:35:06.946042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.946019    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:06.989996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:06.989967    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:09.093777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:09.089667    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:09.308256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:09.307925    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:09.623343 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:09.622795    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:09.623343 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:09.622823    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:09.623343 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:09.622885    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials podName:254dded1-622a-4ddd-9bf7-856b49eb5118 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:17.622867146 +0000 UTC m=+1395.672898376 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118") : secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:10.660090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:10.660052    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:11.729006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:11.728979    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:11.793294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:11.793269    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:35:13.746911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:13.746877    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:13.833408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:13.833377    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:35:13.932153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:13.932121    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"]
Nov 03 14:35:13.966203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:13.966174    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:15.862903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:15.862872    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:15.946295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:15.946270    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:15.958456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:15.958433    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:16.944580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:16.944550    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:17.646119 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:17.646083    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b44368991f6c: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:17.646119 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:17.646111    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w: secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:17.646358 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:17.646175    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials podName:254dded1-622a-4ddd-9bf7-856b49eb5118 nodeName:}" failed. No retries permitted until 2026-11-03 14:35:33.646159063 +0000 UTC m=+1411.696190229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118") : secret "agc-cred-b44368991f6c" not found
Nov 03 14:35:17.908456 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:17.908425    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:18.932902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:18.932864    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" podUID="254dded1-622a-4ddd-9bf7-856b49eb5118" containerName="payload-processing" containerID="cri-o://d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f" gracePeriod=25
Nov 03 14:35:19.199419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.199368    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:35:19.265776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.265747    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials\") pod \"254dded1-622a-4ddd-9bf7-856b49eb5118\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") "
Nov 03 14:35:19.265942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.265824    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt\") pod \"254dded1-622a-4ddd-9bf7-856b49eb5118\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") "
Nov 03 14:35:19.265942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.265854    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume\") pod \"254dded1-622a-4ddd-9bf7-856b49eb5118\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") "
Nov 03 14:35:19.266062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.266024    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume\") pod \"254dded1-622a-4ddd-9bf7-856b49eb5118\" (UID: \"254dded1-622a-4ddd-9bf7-856b49eb5118\") "
Nov 03 14:35:19.266355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.266334    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume" pod "254dded1-622a-4ddd-9bf7-856b49eb5118" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:35:19.266478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.266397    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume" pod "254dded1-622a-4ddd-9bf7-856b49eb5118" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:35:19.266667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.266649    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:19.266736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.266673    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/254dded1-622a-4ddd-9bf7-856b49eb5118-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:19.268078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.268053    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt" pod "254dded1-622a-4ddd-9bf7-856b49eb5118" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118"). InnerVolumeSpecName "kube-api-access-qrzpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:35:19.268287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.268263    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials" pod "254dded1-622a-4ddd-9bf7-856b49eb5118" (UID: "254dded1-622a-4ddd-9bf7-856b49eb5118"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:35:19.368047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.368014    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:19.368047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.368046    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrzpt\" (UniqueName: \"kubernetes.io/projected/254dded1-622a-4ddd-9bf7-856b49eb5118-kube-api-access-qrzpt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:19.893579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.893548    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:19.959701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.959665    6481 generic.go:356] "Generic (PLEG): container finished" podID="254dded1-622a-4ddd-9bf7-856b49eb5118" containerID="d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f" exitCode=0
Nov 03 14:35:19.960155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.959706    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"
Nov 03 14:35:19.960155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.959720    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" event={"ID":"254dded1-622a-4ddd-9bf7-856b49eb5118","Type":"ContainerDied","Data":"d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f"}
Nov 03 14:35:19.960155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.959771    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w" event={"ID":"254dded1-622a-4ddd-9bf7-856b49eb5118","Type":"ContainerDied","Data":"bfb657a35cba7970f609cf1ae42f38155ec64a0fa7f6b6e485baf0e7c6de1ceb"}
Nov 03 14:35:19.960155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.959804    6481 scope.go:122] "RemoveContainer" containerID="d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f"
Nov 03 14:35:19.971440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.971422    6481 scope.go:122] "RemoveContainer" containerID="d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f"
Nov 03 14:35:19.971689 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:19.971669    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f\": container with ID starting with d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f not found: ID does not exist" containerID="d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f"
Nov 03 14:35:19.971732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:19.971697    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f"} err="failed to get container status \"d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f\": rpc error: code = NotFound desc = could not find container \"d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f\": container with ID starting with d7596f5f80c3f852728c8ae96018cb53b3bd24d40cf66ce30c5646d9edde596f not found: ID does not exist"
Nov 03 14:35:20.016127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:20.016095    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"]
Nov 03 14:35:20.027522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:20.027495    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-846ffws8w"]
Nov 03 14:35:20.067983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:20.067957    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254dded1-622a-4ddd-9bf7-856b49eb5118" path="/var/lib/kubelet/pods/254dded1-622a-4ddd-9bf7-856b49eb5118/volumes"
Nov 03 14:35:20.070422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:20.070401    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"]
Nov 03 14:35:20.074038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:20.074018    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5de7cbe-zp6jb"]
Nov 03 14:35:21.138255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:21.137413    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:35:22.188187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:22.187577    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22654ee6-b16f-4add-a55c-cbabe1f6674b" path="/var/lib/kubelet/pods/22654ee6-b16f-4add-a55c-cbabe1f6674b/volumes"
Nov 03 14:35:23.398124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:23.398095    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"]
Nov 03 14:35:23.555634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:23.555573    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:23.781257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:23.781191    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:23.998106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:23.998072    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:24.336794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:24.336763    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:25.154671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:25.154640    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:25.361416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:25.361385    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:26.435912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.435868    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:26.600246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.600194    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"]
Nov 03 14:35:26.600875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.600858    6481 cpu_manager.go:414] "containerMap: removing container" podUID="254dded1-622a-4ddd-9bf7-856b49eb5118" containerName="payload-processing"
Nov 03 14:35:26.600875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.600875    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254dded1-622a-4ddd-9bf7-856b49eb5118" containerName="payload-processing"
Nov 03 14:35:26.601045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.601026    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="254dded1-622a-4ddd-9bf7-856b49eb5118" containerName="payload-processing"
Nov 03 14:35:26.606819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.606800    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.633920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.633872    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 14:35:26.634379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.633979    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-c4fsz\""
Nov 03 14:35:26.634379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.634017    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-cc6f089120a5\""
Nov 03 14:35:26.634379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.634103    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 14:35:26.634379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.634164    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 14:35:26.634379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.634269    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 14:35:26.634674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.634559    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7469958f157e\""
Nov 03 14:35:26.640981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.640953    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"]
Nov 03 14:35:26.705497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.705442    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.705614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.705506    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.705658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.705617    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.705699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.705686    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfskq\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.815265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.815109    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.815476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.815273    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.815476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.815382    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.815604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.815476    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfskq\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.819254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.816807    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.834751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.833167    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.848974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.848943    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfskq\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.860414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.860380    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials\") pod \"payload-processing-external-model-65b496c9b-pbkxk\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") " pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.921073 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.921040    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:26.992733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:26.992641    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:27.308091 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:27.308048    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67dfe17_d1d7_472d_9d6b_2894aaeebc9a.slice/crio-3df85394c89736b75183ec2886e24387232ce48e11ed9a0c6a2b8b3343d410c2 WatchSource:0}: Error finding container 3df85394c89736b75183ec2886e24387232ce48e11ed9a0c6a2b8b3343d410c2: Status 404 returned error can't find the container with id 3df85394c89736b75183ec2886e24387232ce48e11ed9a0c6a2b8b3343d410c2
Nov 03 14:35:27.310260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:27.310213    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"]
Nov 03 14:35:27.996035 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:27.995993    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:35:28.040899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:28.040853    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk" event={"ID":"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a","Type":"ContainerStarted","Data":"58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab"}
Nov 03 14:35:28.041082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:28.040933    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:28.041082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:28.040963    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk" event={"ID":"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a","Type":"ContainerStarted","Data":"3df85394c89736b75183ec2886e24387232ce48e11ed9a0c6a2b8b3343d410c2"}
Nov 03 14:35:28.095669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:28.095619    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk" podStartSLOduration=2.09560676 podStartE2EDuration="2.09560676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:28.090978964 +0000 UTC m=+1406.141010151" watchObservedRunningTime="2026-11-03 14:35:28.09560676 +0000 UTC m=+1406.145637942"
Nov 03 14:35:29.074586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:29.074538    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:29.287320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:29.286709    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:31.259106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:31.259067    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:32.302275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.302222    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"]
Nov 03 14:35:32.306826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.306807    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.331290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.331261    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"]
Nov 03 14:35:32.366104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366063    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn9rk\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366106    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366246    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366315    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366399    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366447    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366548    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366600    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.366680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.366665    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.467795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.467759    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.467865    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.467916    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.467979    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468027    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468091    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468206    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn9rk\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468251    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468268    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468309    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468563    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.468724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.468687    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.469081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.469036    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.469505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.469483    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.470904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.470882    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.471208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.471192    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.493720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.493692    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.493906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.493875    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn9rk\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk\") pod \"e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") " pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:32.622871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:32.622788    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:33.026301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:33.020610    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"]
Nov 03 14:35:33.043082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:33.042905    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:35:33.043224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:33.043140    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:35:33.043224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:33.043195    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:35:33.146266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:33.143533    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" event={"ID":"9afb21a5-fe6f-410a-949b-41ef42ee1ccb","Type":"ContainerStarted","Data":"7814c2c4b92434322f8dd44f64604270de5818a731ce546f39befe46c41f6dcd"}
Nov 03 14:35:33.517288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:33.510398    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:34.163088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:34.163055    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" event={"ID":"9afb21a5-fe6f-410a-949b-41ef42ee1ccb","Type":"ContainerStarted","Data":"6ba5888ab83638817c34d40b1ec860af341203d4d2be76578aa2f1df95b1bcd1"}
Nov 03 14:35:34.627204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:34.627114    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:34.627931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:34.627906    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 14:35:34.628101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:34.628075    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 14:35:34.717746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:34.717716    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:35.426730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.426665    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podStartSLOduration=3.426644991 podStartE2EDuration="3.426644991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:34.226339923 +0000 UTC m=+1412.276371108" watchObservedRunningTime="2026-11-03 14:35:35.426644991 +0000 UTC m=+1413.476676176"
Nov 03 14:35:35.433306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.433269    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"]
Nov 03 14:35:35.453901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.453860    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.477494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.477458    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"]
Nov 03 14:35:35.496595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.478567    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:35:35.501546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.501523    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:35:35.599478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.524576    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:35:35.599710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.524796    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:35:35.599817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.576424    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.599961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.599944    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.600347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.600323    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt99n\" (UniqueName: \"kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.659253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.635603    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 14:35:35.659253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.635674    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 14:35:35.722307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.721706    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.752286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.730288    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.752286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.730487    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt99n\" (UniqueName: \"kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.752286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.731766    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.752286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.740835    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:35.820320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:35.820280    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt99n\" (UniqueName: \"kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n\") pod \"maas-api-revoke-sub-61874855-f5km2\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:36.010306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.006469    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:36.472613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.472573    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:36.585254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.584450    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:35:36.599781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.596328    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:35:36.665675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.665635    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 14:35:36.686343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.686306    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 14:35:36.944643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:36.944517    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" podUID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:35:37.221317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:37.214710    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"]
Nov 03 14:35:37.314251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:37.314188    6481 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:35:37.318248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:37.314785    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Nov 03 14:35:37.341262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:37.339305    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:35:37.407252 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:37.402940    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df9033c_d425_4a69_904a_91ecf27c711f.slice/crio-a0299aae13138c9dbb2a97f9b1c4f49f258514191a455a135930d265f9467300 WatchSource:0}: Error finding container a0299aae13138c9dbb2a97f9b1c4f49f258514191a455a135930d265f9467300: Status 404 returned error can't find the container with id a0299aae13138c9dbb2a97f9b1c4f49f258514191a455a135930d265f9467300
Nov 03 14:35:37.627253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:37.626620    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 14:35:37.627253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:37.626684    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 14:35:38.425755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.425713    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" podUID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" containerName="main" containerID="cri-o://c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6" gracePeriod=45
Nov 03 14:35:38.431955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.431924    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2" event={"ID":"7df9033c-d425-4a69-904a-91ecf27c711f","Type":"ContainerStarted","Data":"1c8ca0063fe228ed6eff218dc1c3b6009bce5fe7eef0f6dff4a0656c496b322b"}
Nov 03 14:35:38.432060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.431978    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2" event={"ID":"7df9033c-d425-4a69-904a-91ecf27c711f","Type":"ContainerStarted","Data":"a0299aae13138c9dbb2a97f9b1c4f49f258514191a455a135930d265f9467300"}
Nov 03 14:35:38.475039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.474996    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2" podStartSLOduration=3.47497788 podStartE2EDuration="3.47497788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:38.473302634 +0000 UTC m=+1416.523333813" watchObservedRunningTime="2026-11-03 14:35:38.47497788 +0000 UTC m=+1416.525009064"
Nov 03 14:35:38.628664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.628626    6481 patch_prober.go:29] interesting pod/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body=
Nov 03 14:35:38.628867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.628691    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused"
Nov 03 14:35:38.782962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.782938    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:38.875016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.874986    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs\") pod \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") "
Nov 03 14:35:38.875189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875030    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache\") pod \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") "
Nov 03 14:35:38.875244 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875189    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir\") pod \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") "
Nov 03 14:35:38.875289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875242    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h\" (UniqueName: \"kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h\") pod \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") "
Nov 03 14:35:38.875289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875270    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home\") pod \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") "
Nov 03 14:35:38.875380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875319    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm\") pod \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\" (UID: \"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0\") "
Nov 03 14:35:38.875438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875388    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache" pod "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" (UID: "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:35:38.875497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875445    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir" pod "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" (UID: "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:35:38.875595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875572    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home" pod "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" (UID: "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:35:38.875892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875863    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:38.875892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875882    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:38.875892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.875891    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:38.877430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.877404    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm" pod "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" (UID: "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:35:38.877518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.877489    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h" pod "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" (UID: "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0"). InnerVolumeSpecName "kube-api-access-jk48h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:35:38.877559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.877510    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs" pod "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" (UID: "1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:35:38.976966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.976894    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:38.976966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.976921    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:38.976966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.976934    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk48h\" (UniqueName: \"kubernetes.io/projected/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0-kube-api-access-jk48h\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:38.997410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:38.997383    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:39.078104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.078077    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:35:39.329310 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.321094    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:39.441047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.441011    6481 generic.go:356] "Generic (PLEG): container finished" podID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" containerID="c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6" exitCode=0
Nov 03 14:35:39.441504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.441063    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"
Nov 03 14:35:39.441504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.441097    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" event={"ID":"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0","Type":"ContainerDied","Data":"c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6"}
Nov 03 14:35:39.441504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.441162    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv" event={"ID":"1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0","Type":"ContainerDied","Data":"b29a01af3c189bc34dfedaa6c796526e28d00b97f4ef6f5c439910a64b2cebac"}
Nov 03 14:35:39.441504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.441187    6481 scope.go:122] "RemoveContainer" containerID="c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6"
Nov 03 14:35:39.443615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.443586    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2" event={"ID":"7df9033c-d425-4a69-904a-91ecf27c711f","Type":"ContainerDied","Data":"1c8ca0063fe228ed6eff218dc1c3b6009bce5fe7eef0f6dff4a0656c496b322b"}
Nov 03 14:35:39.443727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.443589    6481 generic.go:356] "Generic (PLEG): container finished" podID="7df9033c-d425-4a69-904a-91ecf27c711f" containerID="1c8ca0063fe228ed6eff218dc1c3b6009bce5fe7eef0f6dff4a0656c496b322b" exitCode=0
Nov 03 14:35:39.454826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.454806    6481 scope.go:122] "RemoveContainer" containerID="c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6"
Nov 03 14:35:39.455127 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:39.455106    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6\": container with ID starting with c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6 not found: ID does not exist" containerID="c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6"
Nov 03 14:35:39.455174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.455136    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6"} err="failed to get container status \"c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6\": rpc error: code = NotFound desc = could not find container \"c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6\": container with ID starting with c2c6c25b48df82510f571c2be234ca81bd310118d215dc60106fd8da2b0bd7b6 not found: ID does not exist"
Nov 03 14:35:39.495816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.495781    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"]
Nov 03 14:35:39.507392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.507359    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4a0349/explicit-model-4a0349-kserve-c5c45d85c-vpttv"]
Nov 03 14:35:39.699571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.699540    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:39.700222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.700198    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:39.701086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:39.701059    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:35:40.068102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.068024    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" path="/var/lib/kubelet/pods/1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0/volumes"
Nov 03 14:35:40.618165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.618139    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:40.804080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.804000    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca\") pod \"7df9033c-d425-4a69-904a-91ecf27c711f\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") "
Nov 03 14:35:40.804266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.804121    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca\") pod \"7df9033c-d425-4a69-904a-91ecf27c711f\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") "
Nov 03 14:35:40.804266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.804167    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n\" (UniqueName: \"kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n\") pod \"7df9033c-d425-4a69-904a-91ecf27c711f\" (UID: \"7df9033c-d425-4a69-904a-91ecf27c711f\") "
Nov 03 14:35:40.804467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.804442    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca" pod "7df9033c-d425-4a69-904a-91ecf27c711f" (UID: "7df9033c-d425-4a69-904a-91ecf27c711f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:35:40.804800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.804754    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:40.806220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.806197    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n" pod "7df9033c-d425-4a69-904a-91ecf27c711f" (UID: "7df9033c-d425-4a69-904a-91ecf27c711f"). InnerVolumeSpecName "kube-api-access-mt99n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:35:40.806351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.806252    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca" pod "7df9033c-d425-4a69-904a-91ecf27c711f" (UID: "7df9033c-d425-4a69-904a-91ecf27c711f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:35:40.905816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.905783    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt99n\" (UniqueName: \"kubernetes.io/projected/7df9033c-d425-4a69-904a-91ecf27c711f-kube-api-access-mt99n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:40.905816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:40.905810    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df9033c-d425-4a69-904a-91ecf27c711f-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:35:41.043340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:41.043304    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"]
Nov 03 14:35:41.047704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:41.047676    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7733e9cb-fgxsp"]
Nov 03 14:35:41.299528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:41.299498    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:41.461095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:41.461065    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"
Nov 03 14:35:41.461305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:41.461061    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2" event={"ID":"7df9033c-d425-4a69-904a-91ecf27c711f","Type":"ContainerDied","Data":"a0299aae13138c9dbb2a97f9b1c4f49f258514191a455a135930d265f9467300"}
Nov 03 14:35:41.461305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:41.461192    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0299aae13138c9dbb2a97f9b1c4f49f258514191a455a135930d265f9467300"
Nov 03 14:35:42.069779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:42.069745    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb011c3-b2ba-408a-bd16-2cc8497b0ca2" path="/var/lib/kubelet/pods/bbb011c3-b2ba-408a-bd16-2cc8497b0ca2/volumes"
Nov 03 14:35:43.294355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:43.294317    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:44.622738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:44.615514    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:44.831990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:44.831955    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:45.392747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:45.392711    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:35:45.483845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:45.483782    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:46.017722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:46.017681    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:47.164124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:47.164087    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:47.717251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:47.716354    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:49.995825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:49.995791    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:50.704864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:50.704830    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:51.020315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:51.016789    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:52.112027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.111998    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"]
Nov 03 14:35:52.112617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.112601    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7df9033c-d425-4a69-904a-91ecf27c711f" containerName="revoke-keys"
Nov 03 14:35:52.112657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.112618    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df9033c-d425-4a69-904a-91ecf27c711f" containerName="revoke-keys"
Nov 03 14:35:52.112657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.112634    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" containerName="main"
Nov 03 14:35:52.112657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.112640    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" containerName="main"
Nov 03 14:35:52.112765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.112755    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df9033c-d425-4a69-904a-91ecf27c711f" containerName="revoke-keys"
Nov 03 14:35:52.112799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.112766    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa17f2c-0e6d-4e92-ad27-d39398ebc8c0" containerName="main"
Nov 03 14:35:52.116314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.116298    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.137154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.137127    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"]
Nov 03 14:35:52.143198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.143174    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-386f6d33da21\""
Nov 03 14:35:52.227703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.227673    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.227845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.227754    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fqq\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.227899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.227844    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.227938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.227903    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.300361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.300328    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:52.328972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.328945    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fqq\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.329142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.329024    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.329142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.329086    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.329273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.329189    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.329856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.329831    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.329947 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.329835    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.331703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.331679    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.349422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.349365    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fqq\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.429553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.429523    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:52.734405 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:52.734379    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68ba447_f64a_47b0_b4cb_2030752126b1.slice/crio-1b56f58ab983118d01b56c6b2019ece0b0b982541fbb20faa8273eeca7de61d8 WatchSource:0}: Error finding container 1b56f58ab983118d01b56c6b2019ece0b0b982541fbb20faa8273eeca7de61d8: Status 404 returned error can't find the container with id 1b56f58ab983118d01b56c6b2019ece0b0b982541fbb20faa8273eeca7de61d8
Nov 03 14:35:52.734557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:52.734535    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"]
Nov 03 14:35:53.190820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.190783    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"]
Nov 03 14:35:53.195177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.195156    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.217653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.217094    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"]
Nov 03 14:35:53.353785 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.353758    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.353937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.353811    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qlkt\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.354040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.354013    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.354163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.354146    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.470334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.460168    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.494312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.470639    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.494312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.470821    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.494312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.470896    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qlkt\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.494312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.472120    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.504268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.495611    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.504268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.496121    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.519585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.519511    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qlkt\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt\") pod \"payload-processing-external-model-64f7c4d6bf-hddgq\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") " pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.530624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.524673    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:53.642018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.639675    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29" event={"ID":"e68ba447-f64a-47b0-b4cb-2030752126b1","Type":"ContainerStarted","Data":"80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625"}
Nov 03 14:35:53.642018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.639735    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29" event={"ID":"e68ba447-f64a-47b0-b4cb-2030752126b1","Type":"ContainerStarted","Data":"1b56f58ab983118d01b56c6b2019ece0b0b982541fbb20faa8273eeca7de61d8"}
Nov 03 14:35:53.642018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.640800    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:35:53.691008 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.688802    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29" podStartSLOduration=1.68878255 podStartE2EDuration="1.68878255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:53.687622051 +0000 UTC m=+1431.737653236" watchObservedRunningTime="2026-11-03 14:35:53.68878255 +0000 UTC m=+1431.738813734"
Nov 03 14:35:53.945178 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:35:53.945146    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f05d8cc_0daf_4b22_a52a_66307ddd69e9.slice/crio-bb23d93d5477570a460dcdac4d9adb93268d516b2a3420cf744832df2ae4c939 WatchSource:0}: Error finding container bb23d93d5477570a460dcdac4d9adb93268d516b2a3420cf744832df2ae4c939: Status 404 returned error can't find the container with id bb23d93d5477570a460dcdac4d9adb93268d516b2a3420cf744832df2ae4c939
Nov 03 14:35:53.945918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:53.945889    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"]
Nov 03 14:35:54.268337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:54.268221    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:54.664746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:54.664707    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq" event={"ID":"1f05d8cc-0daf-4b22-a52a-66307ddd69e9","Type":"ContainerStarted","Data":"aca68e0f8997bd5e165a10f0c079dff479d674310dbfb29f3591a31e79f04429"}
Nov 03 14:35:54.665046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:54.665022    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq" event={"ID":"1f05d8cc-0daf-4b22-a52a-66307ddd69e9","Type":"ContainerStarted","Data":"bb23d93d5477570a460dcdac4d9adb93268d516b2a3420cf744832df2ae4c939"}
Nov 03 14:35:54.665260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:54.665243    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:35:54.717396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:54.712919    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq" podStartSLOduration=1.7128994629999998 podStartE2EDuration="1.712899463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:54.706046489 +0000 UTC m=+1432.756077673" watchObservedRunningTime="2026-11-03 14:35:54.712899463 +0000 UTC m=+1432.762930646"
Nov 03 14:35:54.790276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:54.789640    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:55.010314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:55.005803    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:56.950265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:56.947867    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:57.986768 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:35:57.986726    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:35:58.887158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:58.887117    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:35:59.117062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:35:59.117033    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:00.692282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:00.692253    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:00.900913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:00.900878    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:01.126474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:01.126397    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:01.191345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:01.191313    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:36:03.013926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:03.013882    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:04.987252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:04.985088    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:05.681967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:05.681933    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:36:05.788646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:05.788617    6481 scope.go:122] "RemoveContainer" containerID="e350848b9673f96b52f67398f74903aaee3e5770b7cf9b2b3e4ba2c6cd52507f"
Nov 03 14:36:05.793245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:05.792994    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"]
Nov 03 14:36:05.886160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:05.886138    6481 scope.go:122] "RemoveContainer" containerID="6cb734cf29238d54a65e43a28a773c2f65f8feffdae638b5f77f0ec3905b2ddf"
Nov 03 14:36:05.926528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:05.910548    6481 scope.go:122] "RemoveContainer" containerID="5677c57d864c782dc9d7da178ea555639b814bda440e8990cd878affe7880cd2"
Nov 03 14:36:06.007153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:06.007123    6481 scope.go:122] "RemoveContainer" containerID="15a850bc8e10b63e4b8bd4b0571f5521a321dfe170ca53790eed788d07ea0599"
Nov 03 14:36:06.141317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:06.141214    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:06.362113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:06.361999    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:06.565910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:06.565867    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:06.736251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:06.735506    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:36:06.836077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:06.836051    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"]
Nov 03 14:36:07.972543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:07.972508    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:08.723860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:08.723827    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:10.690927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:10.690882    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:10.794248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:10.794193    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb" podUID="9a196183-2ef2-4f59-8e83-a4eb267d7b26" containerName="payload-processing" containerID="cri-o://ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584" gracePeriod=25
Nov 03 14:36:11.067431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.067409    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:36:11.206595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.206562    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc\") pod \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") "
Nov 03 14:36:11.206843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.206640    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials\") pod \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") "
Nov 03 14:36:11.206843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.206809    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume\") pod \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") "
Nov 03 14:36:11.206981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.206864    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume\") pod \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\" (UID: \"9a196183-2ef2-4f59-8e83-a4eb267d7b26\") "
Nov 03 14:36:11.207355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.207302    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume" pod "9a196183-2ef2-4f59-8e83-a4eb267d7b26" (UID: "9a196183-2ef2-4f59-8e83-a4eb267d7b26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:11.207671 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.207628    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume" pod "9a196183-2ef2-4f59-8e83-a4eb267d7b26" (UID: "9a196183-2ef2-4f59-8e83-a4eb267d7b26"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:11.207868 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.207849    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:11.209399 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.209378    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc" pod "9a196183-2ef2-4f59-8e83-a4eb267d7b26" (UID: "9a196183-2ef2-4f59-8e83-a4eb267d7b26"). InnerVolumeSpecName "kube-api-access-bb9fc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:11.209513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.209451    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials" pod "9a196183-2ef2-4f59-8e83-a4eb267d7b26" (UID: "9a196183-2ef2-4f59-8e83-a4eb267d7b26"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:11.336253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.330931    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9fc\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-kube-api-access-bb9fc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:11.336253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.330963    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a196183-2ef2-4f59-8e83-a4eb267d7b26-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:11.336253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.330979    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a196183-2ef2-4f59-8e83-a4eb267d7b26-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:11.837070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.837001    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk" podUID="f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" containerName="payload-processing" containerID="cri-o://58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab" gracePeriod=25
Nov 03 14:36:11.877075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.877045    6481 generic.go:356] "Generic (PLEG): container finished" podID="9a196183-2ef2-4f59-8e83-a4eb267d7b26" containerID="ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584" exitCode=0
Nov 03 14:36:11.877204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.877109    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb" event={"ID":"9a196183-2ef2-4f59-8e83-a4eb267d7b26","Type":"ContainerDied","Data":"ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584"}
Nov 03 14:36:11.877204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.877174    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb" event={"ID":"9a196183-2ef2-4f59-8e83-a4eb267d7b26","Type":"ContainerDied","Data":"0eb7fe76c28681e037ecc41f7e4ddb5887871f1945a7b724a4dd9d4be12df957"}
Nov 03 14:36:11.877204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.877114    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"
Nov 03 14:36:11.877204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.877201    6481 scope.go:122] "RemoveContainer" containerID="ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584"
Nov 03 14:36:12.016278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:11.992325    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"]
Nov 03 14:36:12.018813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.018791    6481 scope.go:122] "RemoveContainer" containerID="ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584"
Nov 03 14:36:12.020042 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:12.019933    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584\": container with ID starting with ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584 not found: ID does not exist" containerID="ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584"
Nov 03 14:36:12.020042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.019974    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584"} err="failed to get container status \"ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584\": rpc error: code = NotFound desc = could not find container \"ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584\": container with ID starting with ba31e32509cb1a4c74b761025a93c37d5095ea0ee420cae6cce53d6baa289584 not found: ID does not exist"
Nov 03 14:36:12.044776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.044530    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7d78mw8tb"]
Nov 03 14:36:12.084902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.083698    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a196183-2ef2-4f59-8e83-a4eb267d7b26" path="/var/lib/kubelet/pods/9a196183-2ef2-4f59-8e83-a4eb267d7b26/volumes"
Nov 03 14:36:12.280192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.280160    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:36:12.449434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.449404    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq\") pod \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") "
Nov 03 14:36:12.449638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.449539    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume\") pod \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") "
Nov 03 14:36:12.449710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.449656    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials\") pod \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") "
Nov 03 14:36:12.449777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.449737    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume\") pod \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\" (UID: \"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a\") "
Nov 03 14:36:12.449951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.449922    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume" pod "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" (UID: "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:12.450114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.450088    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume" pod "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" (UID: "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:12.450395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.450377    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:12.450469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.450403    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:12.451649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.451627    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq" pod "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" (UID: "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a"). InnerVolumeSpecName "kube-api-access-gfskq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:12.451757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.451741    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials" pod "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" (UID: "f67dfe17-d1d7-472d-9d6b-2894aaeebc9a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:12.551949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.551912    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfskq\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-kube-api-access-gfskq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:12.551949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.551941    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:12.920487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.914916    6481 generic.go:356] "Generic (PLEG): container finished" podID="f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" containerID="58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab" exitCode=0
Nov 03 14:36:12.920487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.915016    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk" event={"ID":"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a","Type":"ContainerDied","Data":"58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab"}
Nov 03 14:36:12.920487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.915067    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk" event={"ID":"f67dfe17-d1d7-472d-9d6b-2894aaeebc9a","Type":"ContainerDied","Data":"3df85394c89736b75183ec2886e24387232ce48e11ed9a0c6a2b8b3343d410c2"}
Nov 03 14:36:12.920487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.915095    6481 scope.go:122] "RemoveContainer" containerID="58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab"
Nov 03 14:36:12.920487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.915215    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"
Nov 03 14:36:12.932594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.932568    6481 scope.go:122] "RemoveContainer" containerID="58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab"
Nov 03 14:36:12.932915 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:12.932889    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab\": container with ID starting with 58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab not found: ID does not exist" containerID="58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab"
Nov 03 14:36:12.932974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.932926    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab"} err="failed to get container status \"58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab\": rpc error: code = NotFound desc = could not find container \"58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab\": container with ID starting with 58b39ecea99f78b011070033829a04ffb9a23eae5f620cffe7e7b09d70e8d4ab not found: ID does not exist"
Nov 03 14:36:12.935404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.935366    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:12.974520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.974450    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"]
Nov 03 14:36:12.985984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:12.985962    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-65b496c9b-pbkxk"]
Nov 03 14:36:13.186356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:13.186250    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:14.068532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:14.068497    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" path="/var/lib/kubelet/pods/f67dfe17-d1d7-472d-9d6b-2894aaeebc9a/volumes"
Nov 03 14:36:14.237517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:14.237488    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:14.467238 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:14.467202    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:15.054764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.054730    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"]
Nov 03 14:36:15.055404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.055390    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a196183-2ef2-4f59-8e83-a4eb267d7b26" containerName="payload-processing"
Nov 03 14:36:15.055464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.055406    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a196183-2ef2-4f59-8e83-a4eb267d7b26" containerName="payload-processing"
Nov 03 14:36:15.055464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.055430    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" containerName="payload-processing"
Nov 03 14:36:15.055464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.055435    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" containerName="payload-processing"
Nov 03 14:36:15.055569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.055531    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a196183-2ef2-4f59-8e83-a4eb267d7b26" containerName="payload-processing"
Nov 03 14:36:15.055569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.055544    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67dfe17-d1d7-472d-9d6b-2894aaeebc9a" containerName="payload-processing"
Nov 03 14:36:15.060936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.060912    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.104038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.095142    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"]
Nov 03 14:36:15.126637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.126606    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:15.133408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.133382    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7469958f157e\""
Nov 03 14:36:15.188285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.188250    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zv4\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.188446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.188413    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.188544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.188506    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.188767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.188746    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.290321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.290272    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.290535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.290516    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zv4\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.290649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.290634    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.290714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.290695    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.291561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.291529    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.291689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.291585    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.294077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.294051    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.314869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.314796    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zv4\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4\") pod \"payload-processing-external-model-689fc7c65b-m8nsp\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") " pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.329809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.329781    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:15.406393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.406357    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.534070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.534035    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:15.799063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.799028    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:15.800795 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:36:15.800726    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb956f2b6_666b_4863_a4fa_0310177b3410.slice/crio-b039e752be2ae8fdaf77394e45b39e6743b826b952bb4eafd938f476d99ed332 WatchSource:0}: Error finding container b039e752be2ae8fdaf77394e45b39e6743b826b952bb4eafd938f476d99ed332: Status 404 returned error can't find the container with id b039e752be2ae8fdaf77394e45b39e6743b826b952bb4eafd938f476d99ed332
Nov 03 14:36:15.801443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.801420    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"]
Nov 03 14:36:15.976658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.976573    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp" event={"ID":"b956f2b6-666b-4863-a4fa-0310177b3410","Type":"ContainerStarted","Data":"ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a"}
Nov 03 14:36:15.977320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.977296    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:15.977476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:15.977334    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp" event={"ID":"b956f2b6-666b-4863-a4fa-0310177b3410","Type":"ContainerStarted","Data":"b039e752be2ae8fdaf77394e45b39e6743b826b952bb4eafd938f476d99ed332"}
Nov 03 14:36:16.014346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:16.014306    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:16.022349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:16.022302    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp" podStartSLOduration=1.022286914 podStartE2EDuration="1.022286914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:16.018487988 +0000 UTC m=+1454.068519173" watchObservedRunningTime="2026-11-03 14:36:16.022286914 +0000 UTC m=+1454.072318096"
Nov 03 14:36:16.045482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:16.045307    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"]
Nov 03 14:36:16.049373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:16.049313    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71f4382e-4bfp5"]
Nov 03 14:36:16.068470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:16.068441    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5712849-7163-4019-b6d0-ecfdfc96e46c" path="/var/lib/kubelet/pods/e5712849-7163-4019-b6d0-ecfdfc96e46c/volumes"
Nov 03 14:36:16.430690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:16.430658    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:17.050068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.050038    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"]
Nov 03 14:36:17.054290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.054267    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.089623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.089597    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-90ead7998bc6\""
Nov 03 14:36:17.094166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.094140    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"]
Nov 03 14:36:17.117141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.117107    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.117350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.117330    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.117527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.117369    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.118444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.117678    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqbl\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.218736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.218703    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.218736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.218739    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.218962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.218795    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqbl\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.218962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.218876    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.219455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.219433    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.219582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.219559    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.221393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.221373    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.229143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.228940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:17.239740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.239714    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqbl\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.378303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.370616    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:17.440666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.440634    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:17.769373 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:36:17.769343    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2684c5_dcc5_4478_b0e3_20c6617c042b.slice/crio-83ce1c5d43ba11420f23fac101601fefa6f4fc97a0915917e05f26fb32dc9a00 WatchSource:0}: Error finding container 83ce1c5d43ba11420f23fac101601fefa6f4fc97a0915917e05f26fb32dc9a00: Status 404 returned error can't find the container with id 83ce1c5d43ba11420f23fac101601fefa6f4fc97a0915917e05f26fb32dc9a00
Nov 03 14:36:17.769600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.769577    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"]
Nov 03 14:36:17.897149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:17.897110    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:18.002063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.002027    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm" event={"ID":"7d2684c5-dcc5-4478-b0e3-20c6617c042b","Type":"ContainerStarted","Data":"e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552"}
Nov 03 14:36:18.002299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.002117    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm" event={"ID":"7d2684c5-dcc5-4478-b0e3-20c6617c042b","Type":"ContainerStarted","Data":"83ce1c5d43ba11420f23fac101601fefa6f4fc97a0915917e05f26fb32dc9a00"}
Nov 03 14:36:18.002299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.002251    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:18.048283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.048179    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm" podStartSLOduration=1.048166452 podStartE2EDuration="1.048166452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:18.043656889 +0000 UTC m=+1456.093688072" watchObservedRunningTime="2026-11-03 14:36:18.048166452 +0000 UTC m=+1456.098197635"
Nov 03 14:36:18.210580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.210543    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"]
Nov 03 14:36:18.247413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.247384    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"]
Nov 03 14:36:18.252081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.252063    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.270412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.270387    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"]
Nov 03 14:36:18.272109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.272083    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-ee9aa20fbb34\""
Nov 03 14:36:18.373161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.373083    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.373350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.373197    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.373414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.373366    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh87\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.373513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.373427    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.474527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.474495    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.474902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.474587    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh87\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.474902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.474636    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.474902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.474727    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.475355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.475333    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.475421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.475405    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.477055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.477038    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.497275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.497225    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh87\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.567484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.567459    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:18.876317 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:36:18.876207    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51ef23d_a0d0_40da_ba6c_7b513b1ca503.slice/crio-fb2dca2bd97eb06757cc56e623a6b615bb87ae289065e621f9610c5e48d5637b WatchSource:0}: Error finding container fb2dca2bd97eb06757cc56e623a6b615bb87ae289065e621f9610c5e48d5637b: Status 404 returned error can't find the container with id fb2dca2bd97eb06757cc56e623a6b615bb87ae289065e621f9610c5e48d5637b
Nov 03 14:36:18.876577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:18.876553    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"]
Nov 03 14:36:19.011882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:19.011845    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd" event={"ID":"f51ef23d-a0d0-40da-ba6c-7b513b1ca503","Type":"ContainerStarted","Data":"079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a"}
Nov 03 14:36:19.012062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:19.011902    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd" event={"ID":"f51ef23d-a0d0-40da-ba6c-7b513b1ca503","Type":"ContainerStarted","Data":"fb2dca2bd97eb06757cc56e623a6b615bb87ae289065e621f9610c5e48d5637b"}
Nov 03 14:36:19.012062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:19.012048    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:19.054369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:19.054311    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd" podStartSLOduration=1.054297726 podStartE2EDuration="1.054297726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:19.051899347 +0000 UTC m=+1457.101930529" watchObservedRunningTime="2026-11-03 14:36:19.054297726 +0000 UTC m=+1457.104328909"
Nov 03 14:36:21.022366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:21.022332    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:21.695101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:21.695076    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:22.245977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:22.245822    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"]
Nov 03 14:36:22.285376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:22.285347    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"]
Nov 03 14:36:22.546441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:22.546342    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:22.825427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:22.825351    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:23.198078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:23.198038    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:36:24.719855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:24.719819    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:24.944212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:24.944140    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:25.020155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.020042    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm" podUID="7d2684c5-dcc5-4478-b0e3-20c6617c042b" containerName="payload-processing" containerID="cri-o://e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552" gracePeriod=25
Nov 03 14:36:25.042123 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.042093    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"]
Nov 03 14:36:25.055868 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.055841    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d4f417a-7w22m"]
Nov 03 14:36:25.317001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.316955    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:25.414254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.411692    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume\") pod \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") "
Nov 03 14:36:25.414254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.411804    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials\") pod \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") "
Nov 03 14:36:25.474253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.473396    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl\") pod \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") "
Nov 03 14:36:25.474253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.473499    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume\") pod \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\" (UID: \"7d2684c5-dcc5-4478-b0e3-20c6617c042b\") "
Nov 03 14:36:25.477252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.475538    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume" pod "7d2684c5-dcc5-4478-b0e3-20c6617c042b" (UID: "7d2684c5-dcc5-4478-b0e3-20c6617c042b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:25.477252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.475669    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume" pod "7d2684c5-dcc5-4478-b0e3-20c6617c042b" (UID: "7d2684c5-dcc5-4478-b0e3-20c6617c042b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:25.478004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.477956    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:25.478247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.478198    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl" pod "7d2684c5-dcc5-4478-b0e3-20c6617c042b" (UID: "7d2684c5-dcc5-4478-b0e3-20c6617c042b"). InnerVolumeSpecName "kube-api-access-rcqbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:25.484451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.484420    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials" pod "7d2684c5-dcc5-4478-b0e3-20c6617c042b" (UID: "7d2684c5-dcc5-4478-b0e3-20c6617c042b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:25.580027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.579883    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcqbl\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-kube-api-access-rcqbl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:25.580027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.579918    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:25.580027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.579937    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d2684c5-dcc5-4478-b0e3-20c6617c042b-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:25.580027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.579949    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d2684c5-dcc5-4478-b0e3-20c6617c042b-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:25.696035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:25.696009    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:26.033252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.031177    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:26.399222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.399110    6481 generic.go:356] "Generic (PLEG): container finished" podID="7d2684c5-dcc5-4478-b0e3-20c6617c042b" containerID="e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552" exitCode=0
Nov 03 14:36:26.399222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.399193    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"
Nov 03 14:36:26.424254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.400659    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd5027b-34d7-4f38-9bc6-f9c77d868292" path="/var/lib/kubelet/pods/5bd5027b-34d7-4f38-9bc6-f9c77d868292/volumes"
Nov 03 14:36:26.424254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.401335    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm" event={"ID":"7d2684c5-dcc5-4478-b0e3-20c6617c042b","Type":"ContainerDied","Data":"e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552"}
Nov 03 14:36:26.424254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.401389    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm" event={"ID":"7d2684c5-dcc5-4478-b0e3-20c6617c042b","Type":"ContainerDied","Data":"83ce1c5d43ba11420f23fac101601fefa6f4fc97a0915917e05f26fb32dc9a00"}
Nov 03 14:36:26.424254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.401422    6481 scope.go:122] "RemoveContainer" containerID="e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552"
Nov 03 14:36:26.480921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.480892    6481 scope.go:122] "RemoveContainer" containerID="e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552"
Nov 03 14:36:26.481347 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:26.481319    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552\": container with ID starting with e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552 not found: ID does not exist" containerID="e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552"
Nov 03 14:36:26.481558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.481521    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552"} err="failed to get container status \"e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552\": rpc error: code = NotFound desc = could not find container \"e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552\": container with ID starting with e5a64619dfa6ba52a75662d25d108f2d3180228e07a48adf115218a99e438552 not found: ID does not exist"
Nov 03 14:36:26.564312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.564278    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"]
Nov 03 14:36:26.577572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.577544    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694v4lpm"]
Nov 03 14:36:26.627253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.611610    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:26.987741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:26.987712    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:36:27.225914 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:27.219150    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:27.251808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:27.251714    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq" podUID="1cfa9272-8448-43a6-a84e-a20a720d1b61" containerName="payload-processing" containerID="cri-o://5efe450a0f342d54252e771bbfcbae98bb72b5611e8bfa7dff519fe23e30391e" gracePeriod=25
Nov 03 14:36:27.306321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:27.305624    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn" podUID="1445e3be-373b-4729-831c-97f7e437abe2" containerName="payload-pre-processing" containerID="cri-o://274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa" gracePeriod=25
Nov 03 14:36:27.375961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:27.375927    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"]
Nov 03 14:36:27.550068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:27.549869    6481 generic.go:356] "Generic (PLEG): container finished" podID="1cfa9272-8448-43a6-a84e-a20a720d1b61" containerID="5efe450a0f342d54252e771bbfcbae98bb72b5611e8bfa7dff519fe23e30391e" exitCode=0
Nov 03 14:36:27.550068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:27.549995    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq" event={"ID":"1cfa9272-8448-43a6-a84e-a20a720d1b61","Type":"ContainerDied","Data":"5efe450a0f342d54252e771bbfcbae98bb72b5611e8bfa7dff519fe23e30391e"}
Nov 03 14:36:28.078996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.077070    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d2684c5-dcc5-4478-b0e3-20c6617c042b" path="/var/lib/kubelet/pods/7d2684c5-dcc5-4478-b0e3-20c6617c042b/volumes"
Nov 03 14:36:28.106412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.104375    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:36:28.114246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.109657    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:36:28.176833 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:28.176752    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:36:28.316828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.316730    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume\") pod \"1445e3be-373b-4729-831c-97f7e437abe2\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") "
Nov 03 14:36:28.317320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.316825    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8\" (UniqueName: \"kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8\") pod \"1cfa9272-8448-43a6-a84e-a20a720d1b61\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") "
Nov 03 14:36:28.317320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.316910    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb\" (UniqueName: \"kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb\") pod \"1445e3be-373b-4729-831c-97f7e437abe2\" (UID: \"1445e3be-373b-4729-831c-97f7e437abe2\") "
Nov 03 14:36:28.317320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.316999    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume\") pod \"1cfa9272-8448-43a6-a84e-a20a720d1b61\" (UID: \"1cfa9272-8448-43a6-a84e-a20a720d1b61\") "
Nov 03 14:36:28.317972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.317941    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume" pod "1cfa9272-8448-43a6-a84e-a20a720d1b61" (UID: "1cfa9272-8448-43a6-a84e-a20a720d1b61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:28.317972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.317951    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume" pod "1445e3be-373b-4729-831c-97f7e437abe2" (UID: "1445e3be-373b-4729-831c-97f7e437abe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:28.322482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.320722    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb" pod "1445e3be-373b-4729-831c-97f7e437abe2" (UID: "1445e3be-373b-4729-831c-97f7e437abe2"). InnerVolumeSpecName "kube-api-access-4g4jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:28.323007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.322981    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8" pod "1cfa9272-8448-43a6-a84e-a20a720d1b61" (UID: "1cfa9272-8448-43a6-a84e-a20a720d1b61"). InnerVolumeSpecName "kube-api-access-hxgz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:28.418810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.418770    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cfa9272-8448-43a6-a84e-a20a720d1b61-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:28.418810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.418806    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1445e3be-373b-4729-831c-97f7e437abe2-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:28.418810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.418816    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxgz8\" (UniqueName: \"kubernetes.io/projected/1cfa9272-8448-43a6-a84e-a20a720d1b61-kube-api-access-hxgz8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:28.418810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.418828    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g4jb\" (UniqueName: \"kubernetes.io/projected/1445e3be-373b-4729-831c-97f7e437abe2-kube-api-access-4g4jb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:28.497263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.497214    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:28.558830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.558802    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"
Nov 03 14:36:28.559014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.558834    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq" event={"ID":"1cfa9272-8448-43a6-a84e-a20a720d1b61","Type":"ContainerDied","Data":"2f1067c84dde6aad9410dc74bdbfb540733447e43c0a7b3850ce06bf34067ee2"}
Nov 03 14:36:28.559014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.558898    6481 scope.go:122] "RemoveContainer" containerID="5efe450a0f342d54252e771bbfcbae98bb72b5611e8bfa7dff519fe23e30391e"
Nov 03 14:36:28.560312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.560297    6481 generic.go:356] "Generic (PLEG): container finished" podID="1445e3be-373b-4729-831c-97f7e437abe2" containerID="274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa" exitCode=0
Nov 03 14:36:28.560437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.560346    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"
Nov 03 14:36:28.560437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.560376    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn" event={"ID":"1445e3be-373b-4729-831c-97f7e437abe2","Type":"ContainerDied","Data":"274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa"}
Nov 03 14:36:28.560437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.560420    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn" event={"ID":"1445e3be-373b-4729-831c-97f7e437abe2","Type":"ContainerDied","Data":"adf63ce57709350c79704735a7281ed9cb4de6bf4491b29a57d84126322bd003"}
Nov 03 14:36:28.571059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.571039    6481 scope.go:122] "RemoveContainer" containerID="274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa"
Nov 03 14:36:28.593991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.593942    6481 scope.go:122] "RemoveContainer" containerID="274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa"
Nov 03 14:36:28.596559 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:28.596527    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa\": container with ID starting with 274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa not found: ID does not exist" containerID="274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa"
Nov 03 14:36:28.596678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.596558    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa"} err="failed to get container status \"274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa\": rpc error: code = NotFound desc = could not find container \"274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa\": container with ID starting with 274b28b1299b5da3cd0862ba9feb1824297694117b12ee8a6c500ba580a021aa not found: ID does not exist"
Nov 03 14:36:28.617313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.617287    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"]
Nov 03 14:36:28.624678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.624655    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2e8614-589c8b7767-8wmxn"]
Nov 03 14:36:28.659327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.659301    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"]
Nov 03 14:36:28.669366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.669339    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2e8614-6fbcdff689-kbdkq"]
Nov 03 14:36:28.712055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:28.712024    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:29.868423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:29.868392    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:30.022983 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.022952    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:36:30.068613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.068579    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1445e3be-373b-4729-831c-97f7e437abe2" path="/var/lib/kubelet/pods/1445e3be-373b-4729-831c-97f7e437abe2/volumes"
Nov 03 14:36:30.069009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.068996    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cfa9272-8448-43a6-a84e-a20a720d1b61" path="/var/lib/kubelet/pods/1cfa9272-8448-43a6-a84e-a20a720d1b61/volumes"
Nov 03 14:36:30.108491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.108461    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"]
Nov 03 14:36:30.141374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.141297    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:30.371914 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.371885    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:30.704217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:30.704184    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:31.342808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:31.342766    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:32.046668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.046622    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"]
Nov 03 14:36:32.058434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.058406    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f28ce9-d5c9m"]
Nov 03 14:36:32.067890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.067863    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b4d115-e5bc-4231-837e-7e23ba26f435" path="/var/lib/kubelet/pods/f9b4d115-e5bc-4231-837e-7e23ba26f435/volumes"
Nov 03 14:36:32.310958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.310877    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:32.376953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.376915    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq" podUID="1f05d8cc-0daf-4b22-a52a-66307ddd69e9" containerName="payload-processing" containerID="cri-o://aca68e0f8997bd5e165a10f0c079dff479d674310dbfb29f3591a31e79f04429" gracePeriod=25
Nov 03 14:36:32.520644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.520610    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:32.654031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.653996    6481 generic.go:356] "Generic (PLEG): container finished" podID="1f05d8cc-0daf-4b22-a52a-66307ddd69e9" containerID="aca68e0f8997bd5e165a10f0c079dff479d674310dbfb29f3591a31e79f04429" exitCode=0
Nov 03 14:36:32.654407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.654378    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq" event={"ID":"1f05d8cc-0daf-4b22-a52a-66307ddd69e9","Type":"ContainerDied","Data":"aca68e0f8997bd5e165a10f0c079dff479d674310dbfb29f3591a31e79f04429"}
Nov 03 14:36:32.824250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:32.824057    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:36:33.009334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.009300    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials\") pod \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") "
Nov 03 14:36:33.009519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.009415    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt\") pod \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") "
Nov 03 14:36:33.009571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.009540    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume\") pod \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") "
Nov 03 14:36:33.009620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.009574    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume\") pod \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\" (UID: \"1f05d8cc-0daf-4b22-a52a-66307ddd69e9\") "
Nov 03 14:36:33.009982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.009958    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume" pod "1f05d8cc-0daf-4b22-a52a-66307ddd69e9" (UID: "1f05d8cc-0daf-4b22-a52a-66307ddd69e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:33.010082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.009976    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume" pod "1f05d8cc-0daf-4b22-a52a-66307ddd69e9" (UID: "1f05d8cc-0daf-4b22-a52a-66307ddd69e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:33.011664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.011642    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials" pod "1f05d8cc-0daf-4b22-a52a-66307ddd69e9" (UID: "1f05d8cc-0daf-4b22-a52a-66307ddd69e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:33.011757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.011736    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt" pod "1f05d8cc-0daf-4b22-a52a-66307ddd69e9" (UID: "1f05d8cc-0daf-4b22-a52a-66307ddd69e9"). InnerVolumeSpecName "kube-api-access-7qlkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:33.111488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.111455    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:33.111488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.111480    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:33.111488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.111490    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:33.111725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.111499    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qlkt\" (UniqueName: \"kubernetes.io/projected/1f05d8cc-0daf-4b22-a52a-66307ddd69e9-kube-api-access-7qlkt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:33.415171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.415123    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:33.664421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.664386    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"
Nov 03 14:36:33.664623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.664421    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq" event={"ID":"1f05d8cc-0daf-4b22-a52a-66307ddd69e9","Type":"ContainerDied","Data":"bb23d93d5477570a460dcdac4d9adb93268d516b2a3420cf744832df2ae4c939"}
Nov 03 14:36:33.664623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.664499    6481 scope.go:122] "RemoveContainer" containerID="aca68e0f8997bd5e165a10f0c079dff479d674310dbfb29f3591a31e79f04429"
Nov 03 14:36:33.720047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.720020    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"]
Nov 03 14:36:33.731225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:33.731199    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-64f7c4d6bf-hddgq"]
Nov 03 14:36:34.067731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:34.067655    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f05d8cc-0daf-4b22-a52a-66307ddd69e9" path="/var/lib/kubelet/pods/1f05d8cc-0daf-4b22-a52a-66307ddd69e9/volumes"
Nov 03 14:36:35.109029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.108985    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29" podUID="e68ba447-f64a-47b0-b4cb-2030752126b1" containerName="payload-processing" containerID="cri-o://80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625" gracePeriod=25
Nov 03 14:36:35.473623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.473599    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:36:35.582286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.581753    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials\") pod \"e68ba447-f64a-47b0-b4cb-2030752126b1\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") "
Nov 03 14:36:35.582286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.581845    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume\") pod \"e68ba447-f64a-47b0-b4cb-2030752126b1\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") "
Nov 03 14:36:35.582618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.582593    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq\") pod \"e68ba447-f64a-47b0-b4cb-2030752126b1\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") "
Nov 03 14:36:35.582958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.582937    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume\") pod \"e68ba447-f64a-47b0-b4cb-2030752126b1\" (UID: \"e68ba447-f64a-47b0-b4cb-2030752126b1\") "
Nov 03 14:36:35.584096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.584067    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume" pod "e68ba447-f64a-47b0-b4cb-2030752126b1" (UID: "e68ba447-f64a-47b0-b4cb-2030752126b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:35.584222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.584203    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume" pod "e68ba447-f64a-47b0-b4cb-2030752126b1" (UID: "e68ba447-f64a-47b0-b4cb-2030752126b1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:36:35.585474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.585451    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials" pod "e68ba447-f64a-47b0-b4cb-2030752126b1" (UID: "e68ba447-f64a-47b0-b4cb-2030752126b1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:35.585752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.585730    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq" pod "e68ba447-f64a-47b0-b4cb-2030752126b1" (UID: "e68ba447-f64a-47b0-b4cb-2030752126b1"). InnerVolumeSpecName "kube-api-access-f6fqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:36:35.594423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.594312    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:35.684770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.684743    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:35.684770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.684767    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fqq\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-kube-api-access-f6fqq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:35.684993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.684779    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e68ba447-f64a-47b0-b4cb-2030752126b1-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:35.684993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.684788    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e68ba447-f64a-47b0-b4cb-2030752126b1-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:36:35.687273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.687213    6481 generic.go:356] "Generic (PLEG): container finished" podID="e68ba447-f64a-47b0-b4cb-2030752126b1" containerID="80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625" exitCode=0
Nov 03 14:36:35.687377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.687278    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"
Nov 03 14:36:35.687377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.687275    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29" event={"ID":"e68ba447-f64a-47b0-b4cb-2030752126b1","Type":"ContainerDied","Data":"80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625"}
Nov 03 14:36:35.687459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.687402    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29" event={"ID":"e68ba447-f64a-47b0-b4cb-2030752126b1","Type":"ContainerDied","Data":"1b56f58ab983118d01b56c6b2019ece0b0b982541fbb20faa8273eeca7de61d8"}
Nov 03 14:36:35.687459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.687425    6481 scope.go:122] "RemoveContainer" containerID="80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625"
Nov 03 14:36:35.697832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.697815    6481 scope.go:122] "RemoveContainer" containerID="80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625"
Nov 03 14:36:35.698079 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:35.698061    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625\": container with ID starting with 80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625 not found: ID does not exist" containerID="80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625"
Nov 03 14:36:35.698129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.698087    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625"} err="failed to get container status \"80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625\": rpc error: code = NotFound desc = could not find container \"80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625\": container with ID starting with 80526a0bcfb2a873c6b28ef5ad27567b57924fe75340eaa252e6d59e08657625 not found: ID does not exist"
Nov 03 14:36:35.745547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.745523    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"]
Nov 03 14:36:35.749184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:35.749167    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfsfb29"]
Nov 03 14:36:36.068083 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:36.068006    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68ba447-f64a-47b0-b4cb-2030752126b1" path="/var/lib/kubelet/pods/e68ba447-f64a-47b0-b4cb-2030752126b1/volumes"
Nov 03 14:36:37.821046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:37.821013    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:38.849436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:38.849393    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:39.062936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:39.062881    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:39.620362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:39.620334    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:39.928938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:39.928911    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:40.143406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:40.143367    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:40.954252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:40.954206    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:41.158087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:41.158053    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:41.360308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:41.360210    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:42.108619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:42.108585    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:42.327693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:42.327663    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:42.550150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:42.545612    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:43.932961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.932931    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"]
Nov 03 14:36:43.933623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933602    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1cfa9272-8448-43a6-a84e-a20a720d1b61" containerName="payload-processing"
Nov 03 14:36:43.933623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933621    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cfa9272-8448-43a6-a84e-a20a720d1b61" containerName="payload-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933655    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1445e3be-373b-4729-831c-97f7e437abe2" containerName="payload-pre-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933663    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1445e3be-373b-4729-831c-97f7e437abe2" containerName="payload-pre-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933701    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e68ba447-f64a-47b0-b4cb-2030752126b1" containerName="payload-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933708    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68ba447-f64a-47b0-b4cb-2030752126b1" containerName="payload-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933724    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1f05d8cc-0daf-4b22-a52a-66307ddd69e9" containerName="payload-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933731    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f05d8cc-0daf-4b22-a52a-66307ddd69e9" containerName="payload-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933746    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7d2684c5-dcc5-4478-b0e3-20c6617c042b" containerName="payload-processing"
Nov 03 14:36:43.933823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933754    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d2684c5-dcc5-4478-b0e3-20c6617c042b" containerName="payload-processing"
Nov 03 14:36:43.934224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933933    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cfa9272-8448-43a6-a84e-a20a720d1b61" containerName="payload-processing"
Nov 03 14:36:43.934224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933954    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1445e3be-373b-4729-831c-97f7e437abe2" containerName="payload-pre-processing"
Nov 03 14:36:43.934224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933968    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68ba447-f64a-47b0-b4cb-2030752126b1" containerName="payload-processing"
Nov 03 14:36:43.934224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933979    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f05d8cc-0daf-4b22-a52a-66307ddd69e9" containerName="payload-processing"
Nov 03 14:36:43.934224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.933989    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d2684c5-dcc5-4478-b0e3-20c6617c042b" containerName="payload-processing"
Nov 03 14:36:43.938850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.938832    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:43.966410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.966385    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-192c988f839b\""
Nov 03 14:36:43.966624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.966421    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0ecbda063701\""
Nov 03 14:36:43.970416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:43.970392    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"]
Nov 03 14:36:44.001787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.001758    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.001920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.001837    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.002026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.002007    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.002183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.002163    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nssd\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.103130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.103099    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.103130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.103133    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.103443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.103193    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.103443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.103291    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nssd\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.103911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.103891    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.103975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.103950    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.105790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.105768    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.123839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.123820    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nssd\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd\") pod \"payload-processing-external-model-6d8799cd9-tnpcj\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") " pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.274479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.274395    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.330553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.330486    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:44.595967 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:36:44.595941    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843ca342_3e73_4d7b_b8ee_51f635fede2d.slice/crio-053f2a9a10a37bbb7c0f7554ac1886c6c0356467f9c6158f8ef78fddb9d1799e WatchSource:0}: Error finding container 053f2a9a10a37bbb7c0f7554ac1886c6c0356467f9c6158f8ef78fddb9d1799e: Status 404 returned error can't find the container with id 053f2a9a10a37bbb7c0f7554ac1886c6c0356467f9c6158f8ef78fddb9d1799e
Nov 03 14:36:44.596948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.596919    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"]
Nov 03 14:36:44.770782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.770750    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj" event={"ID":"843ca342-3e73-4d7b-b8ee-51f635fede2d","Type":"ContainerStarted","Data":"fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208"}
Nov 03 14:36:44.770938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.770813    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj" event={"ID":"843ca342-3e73-4d7b-b8ee-51f635fede2d","Type":"ContainerStarted","Data":"053f2a9a10a37bbb7c0f7554ac1886c6c0356467f9c6158f8ef78fddb9d1799e"}
Nov 03 14:36:44.770938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.770841    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:44.815842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:44.815793    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj" podStartSLOduration=1.815779156 podStartE2EDuration="1.815779156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:44.813375609 +0000 UTC m=+1482.863406792" watchObservedRunningTime="2026-11-03 14:36:44.815779156 +0000 UTC m=+1482.865810338"
Nov 03 14:36:45.364911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:45.364882    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:36:46.505306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:46.505256    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:47.327695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:47.327661    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:47.594318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:47.593663    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:47.800658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:47.800622    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:48.015909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:48.015872    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:48.040143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:48.040114    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"]
Nov 03 14:36:48.051323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:48.051299    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c471679-lr88j"]
Nov 03 14:36:48.067333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:48.067306    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324" path="/var/lib/kubelet/pods/0c19cb1a-a4af-4ed7-b09d-e76b6cd7c324/volumes"
Nov 03 14:36:48.634827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:48.634793    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:49.937897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:49.937861    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:50.150193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:50.150154    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:50.435752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:50.435710    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:50.812164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:50.812086    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:52.795394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:52.795359    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:53.064258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:53.064168    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:54.794429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:54.794400    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:55.780801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:55.780762    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:36:55.867596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:55.867564    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"]
Nov 03 14:36:56.922906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:56.922868    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:57.993817 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:36:57.993770    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:36:58.891007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:58.890966    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:59.132306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:59.131606    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:36:59.368709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:36:59.368679    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:00.868585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:00.868522    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp" podUID="b956f2b6-666b-4863-a4fa-0310177b3410" containerName="payload-processing" containerID="cri-o://ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a" gracePeriod=25
Nov 03 14:37:01.589385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.589352    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:01.929048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.929020    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:37:01.998442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.998402    6481 generic.go:356] "Generic (PLEG): container finished" podID="b956f2b6-666b-4863-a4fa-0310177b3410" containerID="ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a" exitCode=0
Nov 03 14:37:01.998612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.998444    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"
Nov 03 14:37:01.998612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.998484    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp" event={"ID":"b956f2b6-666b-4863-a4fa-0310177b3410","Type":"ContainerDied","Data":"ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a"}
Nov 03 14:37:01.998612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.998539    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp" event={"ID":"b956f2b6-666b-4863-a4fa-0310177b3410","Type":"ContainerDied","Data":"b039e752be2ae8fdaf77394e45b39e6743b826b952bb4eafd938f476d99ed332"}
Nov 03 14:37:01.998612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:01.998567    6481 scope.go:122] "RemoveContainer" containerID="ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a"
Nov 03 14:37:02.005305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.005275    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4\") pod \"b956f2b6-666b-4863-a4fa-0310177b3410\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") "
Nov 03 14:37:02.005455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.005341    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume\") pod \"b956f2b6-666b-4863-a4fa-0310177b3410\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") "
Nov 03 14:37:02.005455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.005449    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials\") pod \"b956f2b6-666b-4863-a4fa-0310177b3410\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") "
Nov 03 14:37:02.005595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.005486    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume\") pod \"b956f2b6-666b-4863-a4fa-0310177b3410\" (UID: \"b956f2b6-666b-4863-a4fa-0310177b3410\") "
Nov 03 14:37:02.006072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.006042    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume" pod "b956f2b6-666b-4863-a4fa-0310177b3410" (UID: "b956f2b6-666b-4863-a4fa-0310177b3410"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:02.006778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.006751    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume" pod "b956f2b6-666b-4863-a4fa-0310177b3410" (UID: "b956f2b6-666b-4863-a4fa-0310177b3410"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:02.007945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.007913    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4" pod "b956f2b6-666b-4863-a4fa-0310177b3410" (UID: "b956f2b6-666b-4863-a4fa-0310177b3410"). InnerVolumeSpecName "kube-api-access-s7zv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:02.010146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.010122    6481 scope.go:122] "RemoveContainer" containerID="ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a"
Nov 03 14:37:02.010510 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:37:02.010452    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a\": container with ID starting with ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a not found: ID does not exist" containerID="ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a"
Nov 03 14:37:02.010584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.010521    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a"} err="failed to get container status \"ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a\": rpc error: code = NotFound desc = could not find container \"ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a\": container with ID starting with ef8e09188ec44c12abb8119713f5be6031604d606ac85748c766eb324a8fcd4a not found: ID does not exist"
Nov 03 14:37:02.013442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.013417    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials" pod "b956f2b6-666b-4863-a4fa-0310177b3410" (UID: "b956f2b6-666b-4863-a4fa-0310177b3410"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:02.107052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.107021    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zv4\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-kube-api-access-s7zv4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:02.107052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.107049    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:02.107326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.107061    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b956f2b6-666b-4863-a4fa-0310177b3410-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:02.107326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.107073    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b956f2b6-666b-4863-a4fa-0310177b3410-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:02.345071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.345032    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"]
Nov 03 14:37:02.355812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:02.355786    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-689fc7c65b-m8nsp"]
Nov 03 14:37:03.727049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:03.727008    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:03.890538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:03.890505    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:37:04.067414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:04.067336    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b956f2b6-666b-4863-a4fa-0310177b3410" path="/var/lib/kubelet/pods/b956f2b6-666b-4863-a4fa-0310177b3410/volumes"
Nov 03 14:37:05.919196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:05.919153    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:06.208865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:06.208756    6481 scope.go:122] "RemoveContainer" containerID="041471fac8e2ba11030445317a97036a151ffda1c196b670ed69b0e8bf9f6ef5"
Nov 03 14:37:06.282834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:06.282808    6481 scope.go:122] "RemoveContainer" containerID="3f9e6e00fea7c270dbe3dd412647ede3311b85f99f33e181bbc6f15c178e4e36"
Nov 03 14:37:06.362261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:06.353383    6481 scope.go:122] "RemoveContainer" containerID="c84df38834f0fb58af937ba7f20fbd421c4d43ff78a540db9602364a63b0e297"
Nov 03 14:37:06.390415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:06.390392    6481 scope.go:122] "RemoveContainer" containerID="39508baf1b82f7b06bf2e6e108689b290422bbd33c833b51f18b4802cdce6ee9"
Nov 03 14:37:08.314251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:08.313975    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:09.714016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:09.713985    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:09.918874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:09.918844    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:10.892956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:10.892929    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:11.300347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.300315    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"]
Nov 03 14:37:11.301019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.300996    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b956f2b6-666b-4863-a4fa-0310177b3410" containerName="payload-processing"
Nov 03 14:37:11.301019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.301018    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b956f2b6-666b-4863-a4fa-0310177b3410" containerName="payload-processing"
Nov 03 14:37:11.301195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.301178    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b956f2b6-666b-4863-a4fa-0310177b3410" containerName="payload-processing"
Nov 03 14:37:11.307298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.307275    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.326821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.326801    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"]
Nov 03 14:37:11.333060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.333040    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-a7888a513bf2\""
Nov 03 14:37:11.358303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.358277    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.358418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.358339    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bffxr\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.358670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.358638    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.358847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.358817    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.460440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.460410    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.460564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.460465    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.460564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.460541    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.460642 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.460585    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bffxr\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.461259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.461209    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.461259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.461221    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.463250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.463219    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.480938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.480914    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bffxr\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.621190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.621107    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:11.944510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:11.944469    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"]
Nov 03 14:37:12.123701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:12.123666    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg" event={"ID":"24dead9f-0c88-40cb-871f-9f27c683f18e","Type":"ContainerStarted","Data":"3fb635f53376a131d56fa02140b4d4ea808aec5e003be665e407be8bcd573914"}
Nov 03 14:37:12.123835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:12.123727    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg" event={"ID":"24dead9f-0c88-40cb-871f-9f27c683f18e","Type":"ContainerStarted","Data":"8d52ea2c684b72354ce3737ab13c3dff1f95cf68862f7558fd8e23faad6b6662"}
Nov 03 14:37:12.123835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:12.123767    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:12.166263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:12.166196    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg" podStartSLOduration=1.166179111 podStartE2EDuration="1.166179111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:12.165403044 +0000 UTC m=+1510.215434227" watchObservedRunningTime="2026-11-03 14:37:12.166179111 +0000 UTC m=+1510.216210294"
Nov 03 14:37:12.907988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:12.907958    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:14.913623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:14.913596    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:16.486251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:16.485637    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:16.728711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:16.728674    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:17.066320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:17.066290    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:18.041404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:18.041375    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"]
Nov 03 14:37:18.052493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:18.052463    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d378834-hdqjk"]
Nov 03 14:37:18.067833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:18.067796    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1347bde2-d8a0-48f0-9dc7-826ab61048f5" path="/var/lib/kubelet/pods/1347bde2-d8a0-48f0-9dc7-826ab61048f5/volumes"
Nov 03 14:37:19.099356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:19.099320    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:19.923709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:19.923677    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:37:21.409479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.409441    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:21.584483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.584453    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"]
Nov 03 14:37:21.590278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.590223    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.610171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.610146    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"]
Nov 03 14:37:21.618665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.617446    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:37:21.618665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.617459    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:37:21.618665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.617494    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:37:21.618665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.617534    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:37:21.713326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.713214    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.713473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.713419    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.713545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.713487    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwpsw\" (UniqueName: \"kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.814663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.814632    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.814821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.814692    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwpsw\" (UniqueName: \"kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.814821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.814781    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.815417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.815396    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.816990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.816970    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.836411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.836383    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwpsw\" (UniqueName: \"kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw\") pod \"maas-api-revoke-sub-b47599cc-76btz\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:21.909776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:21.909755    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:22.214590 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:37:22.214557    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525546a1_64b1_410b_9d0f_847799da8e90.slice/crio-6ba34c8d27a6296479d6f1cfad17b70c70a9755a2aa91e9f0ad21578cdea3082 WatchSource:0}: Error finding container 6ba34c8d27a6296479d6f1cfad17b70c70a9755a2aa91e9f0ad21578cdea3082: Status 404 returned error can't find the container with id 6ba34c8d27a6296479d6f1cfad17b70c70a9755a2aa91e9f0ad21578cdea3082
Nov 03 14:37:22.214972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:22.214949    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"]
Nov 03 14:37:22.238267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:22.238208    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz" event={"ID":"525546a1-64b1-410b-9d0f-847799da8e90","Type":"ContainerStarted","Data":"6ba34c8d27a6296479d6f1cfad17b70c70a9755a2aa91e9f0ad21578cdea3082"}
Nov 03 14:37:23.134256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:23.134219    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:37:23.214938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:23.214904    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"]
Nov 03 14:37:23.256710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:23.256676    6481 generic.go:356] "Generic (PLEG): container finished" podID="525546a1-64b1-410b-9d0f-847799da8e90" containerID="9e7092408b14a7749879f9e05bbdcc9a7f11659e4909a5f98000c4f9eb2c9fa9" exitCode=0
Nov 03 14:37:23.256895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:23.256823    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz" event={"ID":"525546a1-64b1-410b-9d0f-847799da8e90","Type":"ContainerDied","Data":"9e7092408b14a7749879f9e05bbdcc9a7f11659e4909a5f98000c4f9eb2c9fa9"}
Nov 03 14:37:23.413618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:23.413391    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:24.420025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.420003    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:24.552797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.552717    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw\" (UniqueName: \"kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw\") pod \"525546a1-64b1-410b-9d0f-847799da8e90\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") "
Nov 03 14:37:24.553017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.552993    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca\") pod \"525546a1-64b1-410b-9d0f-847799da8e90\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") "
Nov 03 14:37:24.553080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.553038    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca\") pod \"525546a1-64b1-410b-9d0f-847799da8e90\" (UID: \"525546a1-64b1-410b-9d0f-847799da8e90\") "
Nov 03 14:37:24.553391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.553364    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca" pod "525546a1-64b1-410b-9d0f-847799da8e90" (UID: "525546a1-64b1-410b-9d0f-847799da8e90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:24.553601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.553585    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:24.554795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.554767    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw" pod "525546a1-64b1-410b-9d0f-847799da8e90" (UID: "525546a1-64b1-410b-9d0f-847799da8e90"). InnerVolumeSpecName "kube-api-access-lwpsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:24.554944 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.554926    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca" pod "525546a1-64b1-410b-9d0f-847799da8e90" (UID: "525546a1-64b1-410b-9d0f-847799da8e90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:37:24.654970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.654945    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525546a1-64b1-410b-9d0f-847799da8e90-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:24.654970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:24.654969    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwpsw\" (UniqueName: \"kubernetes.io/projected/525546a1-64b1-410b-9d0f-847799da8e90-kube-api-access-lwpsw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:25.053936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:25.053900    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"]
Nov 03 14:37:25.065441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:25.065414    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057d10ad-gk8g9"]
Nov 03 14:37:25.272415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:25.272387    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"
Nov 03 14:37:25.272415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:25.272396    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz" event={"ID":"525546a1-64b1-410b-9d0f-847799da8e90","Type":"ContainerDied","Data":"6ba34c8d27a6296479d6f1cfad17b70c70a9755a2aa91e9f0ad21578cdea3082"}
Nov 03 14:37:25.272638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:25.272438    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ba34c8d27a6296479d6f1cfad17b70c70a9755a2aa91e9f0ad21578cdea3082"
Nov 03 14:37:25.397509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:25.397431    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:26.068402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:26.068367    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353ecd02-3994-42b2-a4ef-7df261c5f32d" path="/var/lib/kubelet/pods/353ecd02-3994-42b2-a4ef-7df261c5f32d/volumes"
Nov 03 14:37:27.316491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.316463    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:27.530927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.530896    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:27.906316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.906279    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"]
Nov 03 14:37:27.906499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.906475    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy" containerID="cri-o://d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2" gracePeriod=30
Nov 03 14:37:27.917435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.917406    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"]
Nov 03 14:37:27.917632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.917610    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy" containerID="cri-o://161b28006d76bbc748ecacd2a97e5e93de7cea321ee066179e917c54fb08754c" gracePeriod=30
Nov 03 14:37:27.921305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.921263    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"]
Nov 03 14:37:27.921463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.921444    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" containerID="cri-o://1394ac8605926b72897342f7b5903d4bcaa847ba54626e50bafb30e0f156ece9" gracePeriod=30
Nov 03 14:37:27.929504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.929479    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"]
Nov 03 14:37:27.929660 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:27.929642    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy" containerID="cri-o://b491b14049d7496512685a0e617acb78b8e19b7144006b418647e9a2e10e33ef" gracePeriod=30
Nov 03 14:37:27.998641 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:37:27.998605    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:37:28.216135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.216100    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd" podUID="f51ef23d-a0d0-40da-ba6c-7b513b1ca503" containerName="payload-processing" containerID="cri-o://079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a" gracePeriod=25
Nov 03 14:37:28.475336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.475278    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:37:28.610503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.610472    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87\") pod \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") "
Nov 03 14:37:28.610670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.610570    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume\") pod \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") "
Nov 03 14:37:28.610716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.610692    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials\") pod \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") "
Nov 03 14:37:28.610716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.610710    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume\") pod \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\" (UID: \"f51ef23d-a0d0-40da-ba6c-7b513b1ca503\") "
Nov 03 14:37:28.611108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.611080    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume" pod "f51ef23d-a0d0-40da-ba6c-7b513b1ca503" (UID: "f51ef23d-a0d0-40da-ba6c-7b513b1ca503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:28.611295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.611112    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume" pod "f51ef23d-a0d0-40da-ba6c-7b513b1ca503" (UID: "f51ef23d-a0d0-40da-ba6c-7b513b1ca503"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:28.611295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.611286    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:28.611446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.611302    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:28.612617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.612596    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87" pod "f51ef23d-a0d0-40da-ba6c-7b513b1ca503" (UID: "f51ef23d-a0d0-40da-ba6c-7b513b1ca503"). InnerVolumeSpecName "kube-api-access-dlh87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:28.612738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.612702    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials" pod "f51ef23d-a0d0-40da-ba6c-7b513b1ca503" (UID: "f51ef23d-a0d0-40da-ba6c-7b513b1ca503"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:28.713102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.713072    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:28.713102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:28.713101    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh87\" (UniqueName: \"kubernetes.io/projected/f51ef23d-a0d0-40da-ba6c-7b513b1ca503-kube-api-access-dlh87\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:29.302804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.302771    6481 generic.go:356] "Generic (PLEG): container finished" podID="f51ef23d-a0d0-40da-ba6c-7b513b1ca503" containerID="079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a" exitCode=0
Nov 03 14:37:29.303058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.302815    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"
Nov 03 14:37:29.303058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.302855    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd" event={"ID":"f51ef23d-a0d0-40da-ba6c-7b513b1ca503","Type":"ContainerDied","Data":"079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a"}
Nov 03 14:37:29.303058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.302902    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd" event={"ID":"f51ef23d-a0d0-40da-ba6c-7b513b1ca503","Type":"ContainerDied","Data":"fb2dca2bd97eb06757cc56e623a6b615bb87ae289065e621f9610c5e48d5637b"}
Nov 03 14:37:29.303058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.302930    6481 scope.go:122] "RemoveContainer" containerID="079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a"
Nov 03 14:37:29.314510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.314489    6481 scope.go:122] "RemoveContainer" containerID="079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a"
Nov 03 14:37:29.314776 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:37:29.314759    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a\": container with ID starting with 079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a not found: ID does not exist" containerID="079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a"
Nov 03 14:37:29.314828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.314785    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a"} err="failed to get container status \"079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a\": rpc error: code = NotFound desc = could not find container \"079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a\": container with ID starting with 079cae8d7aa1ec4851b9d2f6a1daf7807f7224f8a39cefeae37a4a756008258a not found: ID does not exist"
Nov 03 14:37:29.371354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.371312    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"]
Nov 03 14:37:29.374419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.374394    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8q8gqd"]
Nov 03 14:37:29.400774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:29.400740    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:30.044676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:30.044642    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"]
Nov 03 14:37:30.056552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:30.056523    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba8620d5-p8wll"]
Nov 03 14:37:30.067835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:30.067806    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81dcdac5-82ab-474d-83f9-10e268852533" path="/var/lib/kubelet/pods/81dcdac5-82ab-474d-83f9-10e268852533/volumes"
Nov 03 14:37:30.068213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:30.068199    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51ef23d-a0d0-40da-ba6c-7b513b1ca503" path="/var/lib/kubelet/pods/f51ef23d-a0d0-40da-ba6c-7b513b1ca503/volumes"
Nov 03 14:37:31.397220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:31.397188    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:32.201570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.201537    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:32.405919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.405886    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:32.568208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.568133    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 14:37:32.568208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.568196    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 14:37:32.671472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.671442    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"]
Nov 03 14:37:32.672059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.672044    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f51ef23d-a0d0-40da-ba6c-7b513b1ca503" containerName="payload-processing"
Nov 03 14:37:32.672109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.672060    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51ef23d-a0d0-40da-ba6c-7b513b1ca503" containerName="payload-processing"
Nov 03 14:37:32.672109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.672101    6481 cpu_manager.go:414] "containerMap: removing container" podUID="525546a1-64b1-410b-9d0f-847799da8e90" containerName="revoke-keys"
Nov 03 14:37:32.672109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.672106    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525546a1-64b1-410b-9d0f-847799da8e90" containerName="revoke-keys"
Nov 03 14:37:32.672218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.672208    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="525546a1-64b1-410b-9d0f-847799da8e90" containerName="revoke-keys"
Nov 03 14:37:32.672267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.672224    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51ef23d-a0d0-40da-ba6c-7b513b1ca503" containerName="payload-processing"
Nov 03 14:37:32.677341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.677322    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.695857 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.695836    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"]
Nov 03 14:37:32.703138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.703085    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:37:32.703339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.703289    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:37:32.705012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.704951    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:37:32.705159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.705057    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:37:32.780698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.780669    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.780851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.780776    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.781015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.780992    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvzv\" (UniqueName: \"kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.794125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.794088    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:32.882815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.882740    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvzv\" (UniqueName: \"kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.882815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.882790    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.882991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.882833    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.883603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.883580    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.885131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.885110    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.902874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.902845    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvzv\" (UniqueName: \"kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv\") pod \"maas-api-revoke-sub-7eac3785-zbbxb\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:32.990632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:32.990601    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:33.088247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.082604    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:33.318661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.318622    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:33.348523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.348496    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"]
Nov 03 14:37:33.352613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.352495    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerID="1394ac8605926b72897342f7b5903d4bcaa847ba54626e50bafb30e0f156ece9" exitCode=0
Nov 03 14:37:33.352613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.352566    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" event={"ID":"d5336c1d-06b1-40d0-b17a-b717e3ee753e","Type":"ContainerDied","Data":"1394ac8605926b72897342f7b5903d4bcaa847ba54626e50bafb30e0f156ece9"}
Nov 03 14:37:33.352752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.352616    6481 scope.go:122] "RemoveContainer" containerID="5a672c7976207f789092179921928671c21a257d813c995a0c46cabede8426ee"
Nov 03 14:37:33.354414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.354393    6481 generic.go:356] "Generic (PLEG): container finished" podID="058cc99f-47ec-4285-a436-2095e30669e2" containerID="161b28006d76bbc748ecacd2a97e5e93de7cea321ee066179e917c54fb08754c" exitCode=0
Nov 03 14:37:33.354525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.354469    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" event={"ID":"058cc99f-47ec-4285-a436-2095e30669e2","Type":"ContainerDied","Data":"161b28006d76bbc748ecacd2a97e5e93de7cea321ee066179e917c54fb08754c"}
Nov 03 14:37:33.355808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.355517    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:37:33.356508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.356338    6481 generic.go:356] "Generic (PLEG): container finished" podID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerID="d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2" exitCode=0
Nov 03 14:37:33.356508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.356415    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" event={"ID":"53f39e7c-6b8b-44d4-a160-3c921c2288ac","Type":"ContainerDied","Data":"d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2"}
Nov 03 14:37:33.356508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.356457    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q" event={"ID":"53f39e7c-6b8b-44d4-a160-3c921c2288ac","Type":"ContainerDied","Data":"1f12dc0a8c177945b865f5a32b0aacc43f63e313371c2a6119f7de1f5c303a17"}
Nov 03 14:37:33.359132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.358643    6481 generic.go:356] "Generic (PLEG): container finished" podID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerID="b491b14049d7496512685a0e617acb78b8e19b7144006b418647e9a2e10e33ef" exitCode=0
Nov 03 14:37:33.359132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.358815    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" event={"ID":"154bc5be-af4d-4236-b2af-58c839a2eedf","Type":"ContainerDied","Data":"b491b14049d7496512685a0e617acb78b8e19b7144006b418647e9a2e10e33ef"}
Nov 03 14:37:33.364280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.364224    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:37:33.372196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.371998    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:37:33.374042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.374021    6481 scope.go:122] "RemoveContainer" containerID="d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2"
Nov 03 14:37:33.392603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392573    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.392746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392623    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.392746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392685    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.392746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392722    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392786    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392826    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392848    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392861    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392903    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392932    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392968    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.392996    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393032    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393066    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393072    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393120    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393160    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393204    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393258    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393291    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393326    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393372    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393415    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393448    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393538    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.393579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393585    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket\") pod \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\" (UID: \"53f39e7c-6b8b-44d4-a160-3c921c2288ac\") "
Nov 03 14:37:33.394855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393678    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd\") pod \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\" (UID: \"d5336c1d-06b1-40d0-b17a-b717e3ee753e\") "
Nov 03 14:37:33.394855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393734    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.394855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393799    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx\") pod \"058cc99f-47ec-4285-a436-2095e30669e2\" (UID: \"058cc99f-47ec-4285-a436-2095e30669e2\") "
Nov 03 14:37:33.395127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.395051    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.395127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.395073    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.393161    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.395610    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.395621    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.395804    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.396126    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.396157    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.396355    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.396644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.396496    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:33.397144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.396738    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.397144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.396921    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.397566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.397539    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.397677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.397651    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.397774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.397755    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.398835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.398796    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.399135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.399116    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "kube-api-access-pjwwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.400899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.400830    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:37:33.411330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.411289    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.412073 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.412044    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:37:33.413595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.413114    6481 scope.go:122] "RemoveContainer" containerID="d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2"
Nov 03 14:37:33.413595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.413150    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:37:33.413595 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:37:33.413531    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2\": container with ID starting with d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2 not found: ID does not exist" containerID="d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2"
Nov 03 14:37:33.413595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.413548    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "kube-api-access-8jjpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.413595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.413566    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2"} err="failed to get container status \"d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2\": rpc error: code = NotFound desc = could not find container \"d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2\": container with ID starting with d189848a22db57a2b9fe0a0cb020093fe2d6a74511fec0538d32671f2bb683b2 not found: ID does not exist"
Nov 03 14:37:33.415393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.415372    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token" pod "d5336c1d-06b1-40d0-b17a-b717e3ee753e" (UID: "d5336c1d-06b1-40d0-b17a-b717e3ee753e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.415977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.415952    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "kube-api-access-7s2c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.418898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.418872    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.418984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.418955    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token" pod "058cc99f-47ec-4285-a436-2095e30669e2" (UID: "058cc99f-47ec-4285-a436-2095e30669e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.420558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.420521    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy" pod "53f39e7c-6b8b-44d4-a160-3c921c2288ac" (UID: "53f39e7c-6b8b-44d4-a160-3c921c2288ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.435799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.435747    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:37:33.495855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.495816    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.495896    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.495981    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496042    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496062    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.496295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496162    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496193    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496260    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496311    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496341    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data\") pod \"154bc5be-af4d-4236-b2af-58c839a2eedf\" (UID: \"154bc5be-af4d-4236-b2af-58c839a2eedf\") "
Nov 03 14:37:33.496474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496258    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.496474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496367    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.497113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496752    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:33.497113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.496985    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497151    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497184    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497199    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497213    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497239    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497254    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497268    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jjpd\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-kube-api-access-8jjpd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497281    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497295    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjwwx\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-kube-api-access-pjwwx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497313    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/058cc99f-47ec-4285-a436-2095e30669e2-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497326    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497337    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497348    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/058cc99f-47ec-4285-a436-2095e30669e2-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.497364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497365    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497375    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497387    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/058cc99f-47ec-4285-a436-2095e30669e2-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497398    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497409    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154bc5be-af4d-4236-b2af-58c839a2eedf-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497419    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497430    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497447    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497459    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497470    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497483    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f39e7c-6b8b-44d4-a160-3c921c2288ac-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497495    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497507    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2c8\" (UniqueName: \"kubernetes.io/projected/53f39e7c-6b8b-44d4-a160-3c921c2288ac-kube-api-access-7s2c8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497519    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5336c1d-06b1-40d0-b17a-b717e3ee753e-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497532    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f39e7c-6b8b-44d4-a160-3c921c2288ac-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497545    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5336c1d-06b1-40d0-b17a-b717e3ee753e-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.497557    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/058cc99f-47ec-4285-a436-2095e30669e2-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.502698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.502658    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "kube-api-access-qhjrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.502910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.502883    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:33.503032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.502988    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:37:33.504849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.504816    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy" pod "154bc5be-af4d-4236-b2af-58c839a2eedf" (UID: "154bc5be-af4d-4236-b2af-58c839a2eedf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:33.529317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.529285    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:33.598956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.598920    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.598956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.598961    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjrq\" (UniqueName: \"kubernetes.io/projected/154bc5be-af4d-4236-b2af-58c839a2eedf-kube-api-access-qhjrq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.599165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.598976    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:33.599165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:33.598990    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154bc5be-af4d-4236-b2af-58c839a2eedf-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:34.367444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.367359    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"
Nov 03 14:37:34.367444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.367361    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq" event={"ID":"058cc99f-47ec-4285-a436-2095e30669e2","Type":"ContainerDied","Data":"b39e524a5025ecaa7d157d2457642c76f4c4275a586fc58fbc74af60c523325b"}
Nov 03 14:37:34.367679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.367506    6481 scope.go:122] "RemoveContainer" containerID="161b28006d76bbc748ecacd2a97e5e93de7cea321ee066179e917c54fb08754c"
Nov 03 14:37:34.368523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.368502    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"
Nov 03 14:37:34.370198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.370181    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"
Nov 03 14:37:34.370198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.370185    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw" event={"ID":"154bc5be-af4d-4236-b2af-58c839a2eedf","Type":"ContainerDied","Data":"967c60d64afa20aa7df040cd3de5d3518d163159678a4e90518fa577b70659e7"}
Nov 03 14:37:34.371935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.371915    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"
Nov 03 14:37:34.372041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.371937    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7" event={"ID":"d5336c1d-06b1-40d0-b17a-b717e3ee753e","Type":"ContainerDied","Data":"21b99e43b70e48da255ecd7950768d6f977abf66fae5f87dd3045364efcf74dc"}
Nov 03 14:37:34.373734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.373715    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb" event={"ID":"e4418f37-2100-491d-b101-4839d94d6f43","Type":"ContainerStarted","Data":"e572192d405d55b9af8fb938a78def0412e1d28d9dfbb200bc053045dfa8b06c"}
Nov 03 14:37:34.373850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.373763    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb" event={"ID":"e4418f37-2100-491d-b101-4839d94d6f43","Type":"ContainerStarted","Data":"5c75d37e5900e6b12668965d8977b3ec64606d025503374110b17603d1a18ad4"}
Nov 03 14:37:34.379442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.379426    6481 scope.go:122] "RemoveContainer" containerID="b491b14049d7496512685a0e617acb78b8e19b7144006b418647e9a2e10e33ef"
Nov 03 14:37:34.388956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.388942    6481 scope.go:122] "RemoveContainer" containerID="1394ac8605926b72897342f7b5903d4bcaa847ba54626e50bafb30e0f156ece9"
Nov 03 14:37:34.425855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.425822    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"]
Nov 03 14:37:34.437402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.437379    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-smrcw"]
Nov 03 14:37:34.500939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.500904    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"]
Nov 03 14:37:34.508621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.508599    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-l4pj7"]
Nov 03 14:37:34.545585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.545558    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"]
Nov 03 14:37:34.554505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.554485    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-dqn8q"]
Nov 03 14:37:34.562001 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.561980    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"]
Nov 03 14:37:34.562585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562571    6481 cpu_manager.go:414] "containerMap: removing container" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy"
Nov 03 14:37:34.562635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562586    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy"
Nov 03 14:37:34.562635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562604    6481 cpu_manager.go:414] "containerMap: removing container" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy"
Nov 03 14:37:34.562635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562609    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy"
Nov 03 14:37:34.562635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562630    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy"
Nov 03 14:37:34.562635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562634    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562646    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562651    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562661    6481 cpu_manager.go:414] "containerMap: removing container" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562666    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562740    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562751    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562770    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562777    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" containerName="istio-proxy"
Nov 03 14:37:34.562792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.562785    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="058cc99f-47ec-4285-a436-2095e30669e2" containerName="istio-proxy"
Nov 03 14:37:34.566169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.566153    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.587208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.587187    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"]
Nov 03 14:37:34.596069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.596032    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb" podStartSLOduration=2.596019208 podStartE2EDuration="2.596019208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:34.595488685 +0000 UTC m=+1532.645519867" watchObservedRunningTime="2026-11-03 14:37:34.596019208 +0000 UTC m=+1532.646050395"
Nov 03 14:37:34.612742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.612717    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkt9\" (UniqueName: \"kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.612859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.612846    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.612923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.612901    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.634787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.634731    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"]
Nov 03 14:37:34.640366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.640343    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-twltq"]
Nov 03 14:37:34.713948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.713912    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkt9\" (UniqueName: \"kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.714084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.714031    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.714084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.714057    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.714763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.714739    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.716367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.716345    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.734917 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.734894    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkt9\" (UniqueName: \"kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9\") pod \"maas-api-revoke-sub-e4a67309-k8k8l\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:34.879255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:34.879209    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:35.042640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.042608    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"]
Nov 03 14:37:35.045943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.045921    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ece408-jc7qf"]
Nov 03 14:37:35.194946 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:37:35.194913    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb33e163_d7ca_4dae_81cd_622e0c446d6f.slice/crio-7f5051f039d18b6b35eb594b463b69f282331ac60d807f501aa3f277b59071be WatchSource:0}: Error finding container 7f5051f039d18b6b35eb594b463b69f282331ac60d807f501aa3f277b59071be: Status 404 returned error can't find the container with id 7f5051f039d18b6b35eb594b463b69f282331ac60d807f501aa3f277b59071be
Nov 03 14:37:35.196902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.196877    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"]
Nov 03 14:37:35.383645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.383607    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l" event={"ID":"db33e163-d7ca-4dae-81cd-622e0c446d6f","Type":"ContainerStarted","Data":"7f5051f039d18b6b35eb594b463b69f282331ac60d807f501aa3f277b59071be"}
Nov 03 14:37:35.386047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.386021    6481 generic.go:356] "Generic (PLEG): container finished" podID="e4418f37-2100-491d-b101-4839d94d6f43" containerID="e572192d405d55b9af8fb938a78def0412e1d28d9dfbb200bc053045dfa8b06c" exitCode=0
Nov 03 14:37:35.386183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.386100    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb" event={"ID":"e4418f37-2100-491d-b101-4839d94d6f43","Type":"ContainerDied","Data":"e572192d405d55b9af8fb938a78def0412e1d28d9dfbb200bc053045dfa8b06c"}
Nov 03 14:37:35.697519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:35.697482    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:36.067434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.067408    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058cc99f-47ec-4285-a436-2095e30669e2" path="/var/lib/kubelet/pods/058cc99f-47ec-4285-a436-2095e30669e2/volumes"
Nov 03 14:37:36.067940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.067915    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154bc5be-af4d-4236-b2af-58c839a2eedf" path="/var/lib/kubelet/pods/154bc5be-af4d-4236-b2af-58c839a2eedf/volumes"
Nov 03 14:37:36.068521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.068504    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f39e7c-6b8b-44d4-a160-3c921c2288ac" path="/var/lib/kubelet/pods/53f39e7c-6b8b-44d4-a160-3c921c2288ac/volumes"
Nov 03 14:37:36.069215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.069200    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb" path="/var/lib/kubelet/pods/b5f4fcb2-2166-44d0-9fc5-5c02fc4f39bb/volumes"
Nov 03 14:37:36.069655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.069641    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5336c1d-06b1-40d0-b17a-b717e3ee753e" path="/var/lib/kubelet/pods/d5336c1d-06b1-40d0-b17a-b717e3ee753e/volumes"
Nov 03 14:37:36.395701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.395621    6481 generic.go:356] "Generic (PLEG): container finished" podID="db33e163-d7ca-4dae-81cd-622e0c446d6f" containerID="0683471168b0ee1cc26d56aec2157b733ecc35c14f59cbec416c32cbb0150164" exitCode=0
Nov 03 14:37:36.395855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.395709    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l" event={"ID":"db33e163-d7ca-4dae-81cd-622e0c446d6f","Type":"ContainerDied","Data":"0683471168b0ee1cc26d56aec2157b733ecc35c14f59cbec416c32cbb0150164"}
Nov 03 14:37:36.545637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.545609    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:36.634345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.634316    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca\") pod \"e4418f37-2100-491d-b101-4839d94d6f43\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") "
Nov 03 14:37:36.634529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.634372    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv\" (UniqueName: \"kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv\") pod \"e4418f37-2100-491d-b101-4839d94d6f43\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") "
Nov 03 14:37:36.634529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.634402    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca\") pod \"e4418f37-2100-491d-b101-4839d94d6f43\" (UID: \"e4418f37-2100-491d-b101-4839d94d6f43\") "
Nov 03 14:37:36.634736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.634715    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca" pod "e4418f37-2100-491d-b101-4839d94d6f43" (UID: "e4418f37-2100-491d-b101-4839d94d6f43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:36.634889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.634874    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:36.636412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.636386    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca" pod "e4418f37-2100-491d-b101-4839d94d6f43" (UID: "e4418f37-2100-491d-b101-4839d94d6f43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:37:36.636517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.636473    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv" pod "e4418f37-2100-491d-b101-4839d94d6f43" (UID: "e4418f37-2100-491d-b101-4839d94d6f43"). InnerVolumeSpecName "kube-api-access-2cvzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:36.736564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.736535    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvzv\" (UniqueName: \"kubernetes.io/projected/e4418f37-2100-491d-b101-4839d94d6f43-kube-api-access-2cvzv\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:36.736564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:36.736563    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4418f37-2100-491d-b101-4839d94d6f43-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:37.215698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.215666    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"]
Nov 03 14:37:37.225985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.225958    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"]
Nov 03 14:37:37.403955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.403921    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb" event={"ID":"e4418f37-2100-491d-b101-4839d94d6f43","Type":"ContainerDied","Data":"5c75d37e5900e6b12668965d8977b3ec64606d025503374110b17603d1a18ad4"}
Nov 03 14:37:37.404111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.403981    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c75d37e5900e6b12668965d8977b3ec64606d025503374110b17603d1a18ad4"
Nov 03 14:37:37.404111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.403928    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"
Nov 03 14:37:37.430750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.430722    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:37.557166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.557142    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:37.646747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.646721    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca\") pod \"db33e163-d7ca-4dae-81cd-622e0c446d6f\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") "
Nov 03 14:37:37.646903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.646855    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca\") pod \"db33e163-d7ca-4dae-81cd-622e0c446d6f\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") "
Nov 03 14:37:37.647015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.646991    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9\" (UniqueName: \"kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9\") pod \"db33e163-d7ca-4dae-81cd-622e0c446d6f\" (UID: \"db33e163-d7ca-4dae-81cd-622e0c446d6f\") "
Nov 03 14:37:37.647293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.647265    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca" pod "db33e163-d7ca-4dae-81cd-622e0c446d6f" (UID: "db33e163-d7ca-4dae-81cd-622e0c446d6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:37.647394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.647384    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:37.648911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.648888    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca" pod "db33e163-d7ca-4dae-81cd-622e0c446d6f" (UID: "db33e163-d7ca-4dae-81cd-622e0c446d6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:37:37.649005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.648903    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9" pod "db33e163-d7ca-4dae-81cd-622e0c446d6f" (UID: "db33e163-d7ca-4dae-81cd-622e0c446d6f"). InnerVolumeSpecName "kube-api-access-4hkt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:37.753612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.753571    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db33e163-d7ca-4dae-81cd-622e0c446d6f-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:37.753612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.753616    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkt9\" (UniqueName: \"kubernetes.io/projected/db33e163-d7ca-4dae-81cd-622e0c446d6f-kube-api-access-4hkt9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:37.810278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:37.810187    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:38.189631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:38.189595    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:37:38.413371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:38.413334    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"
Nov 03 14:37:38.413560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:38.413374    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l" event={"ID":"db33e163-d7ca-4dae-81cd-622e0c446d6f","Type":"ContainerDied","Data":"7f5051f039d18b6b35eb594b463b69f282331ac60d807f501aa3f277b59071be"}
Nov 03 14:37:38.413560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:38.413427    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f5051f039d18b6b35eb594b463b69f282331ac60d807f501aa3f277b59071be"
Nov 03 14:37:39.512993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:39.512953    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:39.999461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:39.999430    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:40.207798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:40.207765    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:41.072538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:41.072506    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"]
Nov 03 14:37:41.083885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:41.083861    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9e58365-7f6nj"]
Nov 03 14:37:41.986479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:41.986450    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:42.068013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:42.067986    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea36e045-2271-4ccf-9e5b-074b532e09ce" path="/var/lib/kubelet/pods/ea36e045-2271-4ccf-9e5b-074b532e09ce/volumes"
Nov 03 14:37:43.992777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:43.992704    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:45.451524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:45.451484    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"]
Nov 03 14:37:45.675996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:45.675965    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:45.885288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:45.885194    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:46.111119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.111080    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:46.499750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.499707    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"]
Nov 03 14:37:46.501156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.501124    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e4418f37-2100-491d-b101-4839d94d6f43" containerName="revoke-keys"
Nov 03 14:37:46.501156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.501155    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4418f37-2100-491d-b101-4839d94d6f43" containerName="revoke-keys"
Nov 03 14:37:46.501419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.501182    6481 cpu_manager.go:414] "containerMap: removing container" podUID="db33e163-d7ca-4dae-81cd-622e0c446d6f" containerName="revoke-keys"
Nov 03 14:37:46.501419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.501190    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db33e163-d7ca-4dae-81cd-622e0c446d6f" containerName="revoke-keys"
Nov 03 14:37:46.501515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.501479    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4418f37-2100-491d-b101-4839d94d6f43" containerName="revoke-keys"
Nov 03 14:37:46.501515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.501497    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="db33e163-d7ca-4dae-81cd-622e0c446d6f" containerName="revoke-keys"
Nov 03 14:37:46.507931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.507912    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.522885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.522859    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"]
Nov 03 14:37:46.529282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.529262    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:37:46.529396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.529278    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:37:46.547065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.547037    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:37:46.547187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.547104    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:37:46.657764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.657735    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.657977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.657954    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4fgk\" (UniqueName: \"kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.658054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.658025    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.758681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.758608    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4fgk\" (UniqueName: \"kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.758681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.758656    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.758879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.758707    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.759389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.759369    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.760945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.760923    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.779588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.779564    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4fgk\" (UniqueName: \"kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk\") pod \"maas-api-revoke-sub-c239883d-9mvxf\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.815856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.815831    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:46.822720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.822701    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:46.952021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.951990    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"]
Nov 03 14:37:46.956850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.956832    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:46.979273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:46.979249    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"]
Nov 03 14:37:47.039378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.039301    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"]
Nov 03 14:37:47.043129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.043094    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5deb936d-5zm72"]
Nov 03 14:37:47.061457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.061435    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.061571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.061473    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbsf\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.061649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.061578    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.061777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.061757    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.131404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.131378    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"]
Nov 03 14:37:47.132087 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:37:47.132063    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5135900c_d06b_4479_90c3_3468fdfb5397.slice/crio-f4456e0a1f7d8f7a67725c19d7c876770990e8fd72416c9b9cd6d4ba0d3d10a8 WatchSource:0}: Error finding container f4456e0a1f7d8f7a67725c19d7c876770990e8fd72416c9b9cd6d4ba0d3d10a8: Status 404 returned error can't find the container with id f4456e0a1f7d8f7a67725c19d7c876770990e8fd72416c9b9cd6d4ba0d3d10a8
Nov 03 14:37:47.163024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.162998    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.163139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.163122    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.163196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.163180    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbsf\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.163343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.163316    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.163729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.163710    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.164012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.163993    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.165689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.165667    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.183369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.183345    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbsf\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.271511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.271484    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:47.392872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.392837    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"]
Nov 03 14:37:47.426768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.426745    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"]
Nov 03 14:37:47.506649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.506615    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf" event={"ID":"5135900c-d06b-4479-90c3-3468fdfb5397","Type":"ContainerStarted","Data":"56db2c3ce9623faabc2fe83b1bc7e227c8205e15e1e1aa13a1320fe93d21087c"}
Nov 03 14:37:47.507089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.506670    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf" event={"ID":"5135900c-d06b-4479-90c3-3468fdfb5397","Type":"ContainerStarted","Data":"f4456e0a1f7d8f7a67725c19d7c876770990e8fd72416c9b9cd6d4ba0d3d10a8"}
Nov 03 14:37:47.552594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.552534    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf" podStartSLOduration=1.552514442 podStartE2EDuration="1.552514442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:47.547089576 +0000 UTC m=+1545.597120762" watchObservedRunningTime="2026-11-03 14:37:47.552514442 +0000 UTC m=+1545.602545626"
Nov 03 14:37:47.592591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.592560    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"]
Nov 03 14:37:47.593502 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:37:47.593472    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f62a75a_1223_4f40_bccc_061e8000e822.slice/crio-9be1eb82271bff1713584eee370384afe9f2042a197152d100a2f2380f156497 WatchSource:0}: Error finding container 9be1eb82271bff1713584eee370384afe9f2042a197152d100a2f2380f156497: Status 404 returned error can't find the container with id 9be1eb82271bff1713584eee370384afe9f2042a197152d100a2f2380f156497
Nov 03 14:37:47.627967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.627941    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:47.831212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:47.831178    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:48.130372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.102671    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746ed0da-7bc4-4378-93ae-5320e2f1f5fb" path="/var/lib/kubelet/pods/746ed0da-7bc4-4378-93ae-5320e2f1f5fb/volumes"
Nov 03 14:37:48.515754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.515720    6481 generic.go:356] "Generic (PLEG): container finished" podID="5135900c-d06b-4479-90c3-3468fdfb5397" containerID="56db2c3ce9623faabc2fe83b1bc7e227c8205e15e1e1aa13a1320fe93d21087c" exitCode=0
Nov 03 14:37:48.516168 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.515806    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf" event={"ID":"5135900c-d06b-4479-90c3-3468fdfb5397","Type":"ContainerDied","Data":"56db2c3ce9623faabc2fe83b1bc7e227c8205e15e1e1aa13a1320fe93d21087c"}
Nov 03 14:37:48.517445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.517422    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w" event={"ID":"7f62a75a-1223-4f40-bccc-061e8000e822","Type":"ContainerStarted","Data":"dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd"}
Nov 03 14:37:48.517574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.517464    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w" event={"ID":"7f62a75a-1223-4f40-bccc-061e8000e822","Type":"ContainerStarted","Data":"9be1eb82271bff1713584eee370384afe9f2042a197152d100a2f2380f156497"}
Nov 03 14:37:48.517574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.517486    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:48.616022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.615956    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w" podStartSLOduration=2.615936717 podStartE2EDuration="2.615936717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:48.612258246 +0000 UTC m=+1546.662289445" watchObservedRunningTime="2026-11-03 14:37:48.615936717 +0000 UTC m=+1546.665967902"
Nov 03 14:37:48.619322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.619291    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:48.921890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:48.921851    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:49.681695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.681666    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:49.813998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.813919    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk\" (UniqueName: \"kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk\") pod \"5135900c-d06b-4479-90c3-3468fdfb5397\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") "
Nov 03 14:37:49.814167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.814134    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca\") pod \"5135900c-d06b-4479-90c3-3468fdfb5397\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") "
Nov 03 14:37:49.814290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.814272    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca\") pod \"5135900c-d06b-4479-90c3-3468fdfb5397\" (UID: \"5135900c-d06b-4479-90c3-3468fdfb5397\") "
Nov 03 14:37:49.814551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.814524    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca" pod "5135900c-d06b-4479-90c3-3468fdfb5397" (UID: "5135900c-d06b-4479-90c3-3468fdfb5397"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:49.814728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.814710    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:49.816093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.816073    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk" pod "5135900c-d06b-4479-90c3-3468fdfb5397" (UID: "5135900c-d06b-4479-90c3-3468fdfb5397"). InnerVolumeSpecName "kube-api-access-c4fgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:49.816146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.816110    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca" pod "5135900c-d06b-4479-90c3-3468fdfb5397" (UID: "5135900c-d06b-4479-90c3-3468fdfb5397"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:37:49.916267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.916211    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5135900c-d06b-4479-90c3-3468fdfb5397-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:49.916267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:49.916267    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4fgk\" (UniqueName: \"kubernetes.io/projected/5135900c-d06b-4479-90c3-3468fdfb5397-kube-api-access-c4fgk\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:50.452407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.452367    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj" podUID="843ca342-3e73-4d7b-b8ee-51f635fede2d" containerName="payload-processing" containerID="cri-o://fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208" gracePeriod=25
Nov 03 14:37:50.533663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.533630    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf" event={"ID":"5135900c-d06b-4479-90c3-3468fdfb5397","Type":"ContainerDied","Data":"f4456e0a1f7d8f7a67725c19d7c876770990e8fd72416c9b9cd6d4ba0d3d10a8"}
Nov 03 14:37:50.533663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.533664    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"
Nov 03 14:37:50.533838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.533688    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4456e0a1f7d8f7a67725c19d7c876770990e8fd72416c9b9cd6d4ba0d3d10a8"
Nov 03 14:37:50.701896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.701864    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:50.738132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.738105    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:37:50.825776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.825741    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials\") pod \"843ca342-3e73-4d7b-b8ee-51f635fede2d\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") "
Nov 03 14:37:50.826471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.826446    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd\") pod \"843ca342-3e73-4d7b-b8ee-51f635fede2d\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") "
Nov 03 14:37:50.827079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.827061    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume\") pod \"843ca342-3e73-4d7b-b8ee-51f635fede2d\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") "
Nov 03 14:37:50.827701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.827579    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume" pod "843ca342-3e73-4d7b-b8ee-51f635fede2d" (UID: "843ca342-3e73-4d7b-b8ee-51f635fede2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:50.828209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.828172    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume" pod "843ca342-3e73-4d7b-b8ee-51f635fede2d" (UID: "843ca342-3e73-4d7b-b8ee-51f635fede2d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:37:50.828364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.828347    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume\") pod \"843ca342-3e73-4d7b-b8ee-51f635fede2d\" (UID: \"843ca342-3e73-4d7b-b8ee-51f635fede2d\") "
Nov 03 14:37:50.829069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.829020    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:50.829069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.829045    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/843ca342-3e73-4d7b-b8ee-51f635fede2d-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:50.862497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.853141    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials" pod "843ca342-3e73-4d7b-b8ee-51f635fede2d" (UID: "843ca342-3e73-4d7b-b8ee-51f635fede2d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:50.862497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.855455    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd" pod "843ca342-3e73-4d7b-b8ee-51f635fede2d" (UID: "843ca342-3e73-4d7b-b8ee-51f635fede2d"). InnerVolumeSpecName "kube-api-access-7nssd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:50.930162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.930119    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:50.930162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:50.930160    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nssd\" (UniqueName: \"kubernetes.io/projected/843ca342-3e73-4d7b-b8ee-51f635fede2d-kube-api-access-7nssd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:51.543716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.543680    6481 generic.go:356] "Generic (PLEG): container finished" podID="843ca342-3e73-4d7b-b8ee-51f635fede2d" containerID="fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208" exitCode=0
Nov 03 14:37:51.543918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.543739    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"
Nov 03 14:37:51.543918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.543773    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj" event={"ID":"843ca342-3e73-4d7b-b8ee-51f635fede2d","Type":"ContainerDied","Data":"fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208"}
Nov 03 14:37:51.543918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.543842    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj" event={"ID":"843ca342-3e73-4d7b-b8ee-51f635fede2d","Type":"ContainerDied","Data":"053f2a9a10a37bbb7c0f7554ac1886c6c0356467f9c6158f8ef78fddb9d1799e"}
Nov 03 14:37:51.543918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.543879    6481 scope.go:122] "RemoveContainer" containerID="fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208"
Nov 03 14:37:51.556803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.556782    6481 scope.go:122] "RemoveContainer" containerID="fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208"
Nov 03 14:37:51.557156 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:37:51.557124    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208\": container with ID starting with fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208 not found: ID does not exist" containerID="fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208"
Nov 03 14:37:51.557312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.557167    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208"} err="failed to get container status \"fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208\": rpc error: code = NotFound desc = could not find container \"fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208\": container with ID starting with fab5d895071f2bad7382ef4401e0a401a5dcf1703022645c537ff4ac70ea5208 not found: ID does not exist"
Nov 03 14:37:51.601219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.601186    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"]
Nov 03 14:37:51.609937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:51.609910    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d8799cd9-tnpcj"]
Nov 03 14:37:52.073392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.073352    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843ca342-3e73-4d7b-b8ee-51f635fede2d" path="/var/lib/kubelet/pods/843ca342-3e73-4d7b-b8ee-51f635fede2d/volumes"
Nov 03 14:37:52.254251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.252304    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" containerID="cri-o://fa8f4db23ce63c949329b19dff3919b39ed77bf1974117d7f281911ba0177c41" gracePeriod=45
Nov 03 14:37:52.254251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.253341    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" containerID="cri-o://7bef5cd9a2790019a9ea48be993d280757119001888716148f6f43c0f33b103d" gracePeriod=45
Nov 03 14:37:52.321279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.319517    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": dial tcp 10.133.0.55:8000: connect: connection refused"
Nov 03 14:37:52.393297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.393266    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerName="payload-processing" containerID="cri-o://dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2" gracePeriod=25
Nov 03 14:37:52.431260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.428295    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerName="payload-pre-processing" containerID="cri-o://2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8" gracePeriod=25
Nov 03 14:37:52.597681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.597550    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": dial tcp 10.133.0.103:8000: connect: connection refused"
Nov 03 14:37:52.789279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.787427    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerID="fa8f4db23ce63c949329b19dff3919b39ed77bf1974117d7f281911ba0177c41" exitCode=0
Nov 03 14:37:52.789279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.787608    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" event={"ID":"d5f1f415-bdd9-4a58-82ca-63ac582267f2","Type":"ContainerDied","Data":"fa8f4db23ce63c949329b19dff3919b39ed77bf1974117d7f281911ba0177c41"}
Nov 03 14:37:52.791677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.791647    6481 generic.go:356] "Generic (PLEG): container finished" podID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerID="7bef5cd9a2790019a9ea48be993d280757119001888716148f6f43c0f33b103d" exitCode=0
Nov 03 14:37:52.792007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.791982    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" event={"ID":"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2","Type":"ContainerDied","Data":"7bef5cd9a2790019a9ea48be993d280757119001888716148f6f43c0f33b103d"}
Nov 03 14:37:52.998338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:52.998303    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:37:53.005163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.005136    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:37:53.086187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.068970    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir\") pod \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") "
Nov 03 14:37:53.086856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.086833    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache\") pod \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") "
Nov 03 14:37:53.087298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.087275    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5\" (UniqueName: \"kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5\") pod \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") "
Nov 03 14:37:53.087865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.087846    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs\") pod \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") "
Nov 03 14:37:53.088282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.086736    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir" pod "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" (UID: "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.088435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.087194    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache" pod "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" (UID: "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.088601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.088505    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home" pod "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" (UID: "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.088601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.088318    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home\") pod \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") "
Nov 03 14:37:53.088708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.088599    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm\") pod \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\" (UID: \"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2\") "
Nov 03 14:37:53.089005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.088985    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.089084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.089009    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.089084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.089022    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.091915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.091883    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5" pod "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" (UID: "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2"). InnerVolumeSpecName "kube-api-access-bf5s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:53.092172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.092158    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm" pod "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" (UID: "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.092409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.092388    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs" pod "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" (UID: "cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:37:53.190251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.190112    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir\") pod \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") "
Nov 03 14:37:53.190543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.190482    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir" pod "d5f1f415-bdd9-4a58-82ca-63ac582267f2" (UID: "d5f1f415-bdd9-4a58-82ca-63ac582267f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.190971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.190949    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k\" (UniqueName: \"kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k\") pod \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") "
Nov 03 14:37:53.192502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.192456    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home" pod "d5f1f415-bdd9-4a58-82ca-63ac582267f2" (UID: "d5f1f415-bdd9-4a58-82ca-63ac582267f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.193105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.193073    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home\") pod \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") "
Nov 03 14:37:53.193585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.193565    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k" pod "d5f1f415-bdd9-4a58-82ca-63ac582267f2" (UID: "d5f1f415-bdd9-4a58-82ca-63ac582267f2"). InnerVolumeSpecName "kube-api-access-kmw2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:37:53.194120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.193748    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm\") pod \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") "
Nov 03 14:37:53.194518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.194499    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache\") pod \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") "
Nov 03 14:37:53.195409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.195266    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache" pod "d5f1f415-bdd9-4a58-82ca-63ac582267f2" (UID: "d5f1f415-bdd9-4a58-82ca-63ac582267f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.196056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.196035    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm" pod "d5f1f415-bdd9-4a58-82ca-63ac582267f2" (UID: "d5f1f415-bdd9-4a58-82ca-63ac582267f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:37:53.196262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.195376    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs\") pod \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\" (UID: \"d5f1f415-bdd9-4a58-82ca-63ac582267f2\") "
Nov 03 14:37:53.197629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.197607    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs" pod "d5f1f415-bdd9-4a58-82ca-63ac582267f2" (UID: "d5f1f415-bdd9-4a58-82ca-63ac582267f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:37:53.198221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198199    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198261    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198278    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198293    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198307    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf5s5\" (UniqueName: \"kubernetes.io/projected/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-kube-api-access-bf5s5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198320    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5f1f415-bdd9-4a58-82ca-63ac582267f2-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198334    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198347    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmw2k\" (UniqueName: \"kubernetes.io/projected/d5f1f415-bdd9-4a58-82ca-63ac582267f2-kube-api-access-kmw2k\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.198370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.198361    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:37:53.660306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.656468    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:53.968186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.968096    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:53.990300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.990264    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"
Nov 03 14:37:53.990528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.990489    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg" event={"ID":"cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2","Type":"ContainerDied","Data":"72e5c62264765f0bde283c23ec6410f4c9a14eab273eb804b026386612a1ad87"}
Nov 03 14:37:53.990637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.990547    6481 scope.go:122] "RemoveContainer" containerID="7bef5cd9a2790019a9ea48be993d280757119001888716148f6f43c0f33b103d"
Nov 03 14:37:53.992955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.992927    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls" event={"ID":"d5f1f415-bdd9-4a58-82ca-63ac582267f2","Type":"ContainerDied","Data":"7aac73a6e04aca3cbdd0c257897a0384bef217fed81e14dfbf2c8a626d7358bd"}
Nov 03 14:37:53.993084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:53.993025    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"
Nov 03 14:37:54.037255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.017388    6481 scope.go:122] "RemoveContainer" containerID="fa8f4db23ce63c949329b19dff3919b39ed77bf1974117d7f281911ba0177c41"
Nov 03 14:37:54.094276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.094211    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"]
Nov 03 14:37:54.099322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.099296    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/distinct-w0-86c2e2-kserve-7f9469b7d5-t2qls"]
Nov 03 14:37:54.183346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.183314    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"]
Nov 03 14:37:54.187502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.187467    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-86c2e2/premium-w0-86c2e2-kserve-5995d88d58-vg9dg"]
Nov 03 14:37:54.623258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.614151    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:37:54.642033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:54.641993    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:37:55.213015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:55.212982    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:56.066146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:56.066112    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:56.067563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:56.067541    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" path="/var/lib/kubelet/pods/cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2/volumes"
Nov 03 14:37:56.067960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:56.067946    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" path="/var/lib/kubelet/pods/d5f1f415-bdd9-4a58-82ca-63ac582267f2/volumes"
Nov 03 14:37:56.767414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:56.767387    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:57.047803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.047722    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"]
Nov 03 14:37:57.056892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.056868    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa57fefb-4gcmj"]
Nov 03 14:37:57.548975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.548937    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"]
Nov 03 14:37:57.549841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549807    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5135900c-d06b-4479-90c3-3468fdfb5397" containerName="revoke-keys"
Nov 03 14:37:57.549841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549844    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5135900c-d06b-4479-90c3-3468fdfb5397" containerName="revoke-keys"
Nov 03 14:37:57.549950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549881    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main"
Nov 03 14:37:57.549950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549889    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main"
Nov 03 14:37:57.549950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549913    6481 cpu_manager.go:414] "containerMap: removing container" podUID="843ca342-3e73-4d7b-b8ee-51f635fede2d" containerName="payload-processing"
Nov 03 14:37:57.549950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549921    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="843ca342-3e73-4d7b-b8ee-51f635fede2d" containerName="payload-processing"
Nov 03 14:37:57.550070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549953    6481 cpu_manager.go:414] "containerMap: removing container" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main"
Nov 03 14:37:57.550070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.549960    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main"
Nov 03 14:37:57.550127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.550087    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5135900c-d06b-4479-90c3-3468fdfb5397" containerName="revoke-keys"
Nov 03 14:37:57.550127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.550102    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5f1f415-bdd9-4a58-82ca-63ac582267f2" containerName="main"
Nov 03 14:37:57.550127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.550113    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="843ca342-3e73-4d7b-b8ee-51f635fede2d" containerName="payload-processing"
Nov 03 14:37:57.550127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.550124    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb63f6ab-d548-4c8f-8ff7-0615ca78b4a2" containerName="main"
Nov 03 14:37:57.555384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.555364    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.572074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.572050    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"]
Nov 03 14:37:57.584262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.579641    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:37:57.584262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.579791    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:37:57.584262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.579828    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:37:57.584262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.581976    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:37:57.682978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.682948    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.683151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.682993    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7fx\" (UniqueName: \"kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.683151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.683113    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.784134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.784105    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7fx\" (UniqueName: \"kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.784519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.784171    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.794075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.784756    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.794075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.785606    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.795097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.795072    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.805046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.804990    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7fx\" (UniqueName: \"kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx\") pod \"maas-api-revoke-sub-4a89f400-4bwbm\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.868662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:57.868635    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:37:57.993068 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:37:57.993033    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:37:58.042515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:58.042482    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"]
Nov 03 14:37:58.046729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:58.046707    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb85696a-mrnl5"]
Nov 03 14:37:58.067824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:58.067766    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="843cfeea-9845-4bfc-97fd-03d0cc71cf3a" path="/var/lib/kubelet/pods/843cfeea-9845-4bfc-97fd-03d0cc71cf3a/volumes"
Nov 03 14:37:58.068223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:58.068210    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980497e0-01b8-4fb0-8810-adf0f8eabe38" path="/var/lib/kubelet/pods/980497e0-01b8-4fb0-8810-adf0f8eabe38/volumes"
Nov 03 14:37:58.180669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:58.180644    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"]
Nov 03 14:37:58.181125 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:37:58.181094    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445bc7c6_b5d1_4060_94da_77c8ec523f7b.slice/crio-12718da974a133bf5af00b8569d21979ce896428ffa120f869c601bebe2c9293 WatchSource:0}: Error finding container 12718da974a133bf5af00b8569d21979ce896428ffa120f869c601bebe2c9293: Status 404 returned error can't find the container with id 12718da974a133bf5af00b8569d21979ce896428ffa120f869c601bebe2c9293
Nov 03 14:37:58.901446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:58.901368    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:59.035031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:59.034989    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm" event={"ID":"445bc7c6-b5d1-4060-94da-77c8ec523f7b","Type":"ContainerStarted","Data":"c75614a49fd6b5a0e448d6e7c876eedb3c58b071d9ac5f7d60cd22075145338b"}
Nov 03 14:37:59.035223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:59.035062    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm" event={"ID":"445bc7c6-b5d1-4060-94da-77c8ec523f7b","Type":"ContainerStarted","Data":"12718da974a133bf5af00b8569d21979ce896428ffa120f869c601bebe2c9293"}
Nov 03 14:37:59.079213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:59.079159    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm" podStartSLOduration=2.079144272 podStartE2EDuration="2.079144272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:59.075935226 +0000 UTC m=+1557.125966410" watchObservedRunningTime="2026-11-03 14:37:59.079144272 +0000 UTC m=+1557.129175454"
Nov 03 14:37:59.262214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:59.262181    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:37:59.528577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:59.528504    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:37:59.607434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:37:59.607401    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"]
Nov 03 14:38:00.044686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:00.044655    6481 generic.go:356] "Generic (PLEG): container finished" podID="445bc7c6-b5d1-4060-94da-77c8ec523f7b" containerID="c75614a49fd6b5a0e448d6e7c876eedb3c58b071d9ac5f7d60cd22075145338b" exitCode=0
Nov 03 14:38:00.045098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:00.044748    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm" event={"ID":"445bc7c6-b5d1-4060-94da-77c8ec523f7b","Type":"ContainerDied","Data":"c75614a49fd6b5a0e448d6e7c876eedb3c58b071d9ac5f7d60cd22075145338b"}
Nov 03 14:38:00.923942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:00.923913    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:01.192281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.192213    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:38:01.226266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.226241    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca\") pod \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") "
Nov 03 14:38:01.226391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.226305    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx\" (UniqueName: \"kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx\") pod \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") "
Nov 03 14:38:01.226391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.226370    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca\") pod \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\" (UID: \"445bc7c6-b5d1-4060-94da-77c8ec523f7b\") "
Nov 03 14:38:01.226553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.226525    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca" pod "445bc7c6-b5d1-4060-94da-77c8ec523f7b" (UID: "445bc7c6-b5d1-4060-94da-77c8ec523f7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:01.226819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.226801    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:01.228386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.228366    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca" pod "445bc7c6-b5d1-4060-94da-77c8ec523f7b" (UID: "445bc7c6-b5d1-4060-94da-77c8ec523f7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:38:01.228386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.228390    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx" pod "445bc7c6-b5d1-4060-94da-77c8ec523f7b" (UID: "445bc7c6-b5d1-4060-94da-77c8ec523f7b"). InnerVolumeSpecName "kube-api-access-4j7fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:01.328363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.328337    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7fx\" (UniqueName: \"kubernetes.io/projected/445bc7c6-b5d1-4060-94da-77c8ec523f7b-kube-api-access-4j7fx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:01.328363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.328362    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/445bc7c6-b5d1-4060-94da-77c8ec523f7b-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:01.644012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:01.643931    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:02.065409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:02.065380    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"
Nov 03 14:38:02.069332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:02.069300    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm" event={"ID":"445bc7c6-b5d1-4060-94da-77c8ec523f7b","Type":"ContainerDied","Data":"12718da974a133bf5af00b8569d21979ce896428ffa120f869c601bebe2c9293"}
Nov 03 14:38:02.069504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:02.069344    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12718da974a133bf5af00b8569d21979ce896428ffa120f869c601bebe2c9293"
Nov 03 14:38:03.012781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:03.012724    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:03.224651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:03.224619    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:03.489224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:03.489197    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:03.689900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:03.689869    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:38:04.537808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:04.537765    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:38:04.538291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:04.537777    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:38:04.608643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:04.608605    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg" podUID="24dead9f-0c88-40cb-871f-9f27c683f18e" containerName="payload-processing" containerID="cri-o://3fb635f53376a131d56fa02140b4d4ea808aec5e003be665e407be8bcd573914" gracePeriod=25
Nov 03 14:38:04.815316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:04.815214    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:05.085378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.085304    6481 generic.go:356] "Generic (PLEG): container finished" podID="24dead9f-0c88-40cb-871f-9f27c683f18e" containerID="3fb635f53376a131d56fa02140b4d4ea808aec5e003be665e407be8bcd573914" exitCode=0
Nov 03 14:38:05.085378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.085353    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg" event={"ID":"24dead9f-0c88-40cb-871f-9f27c683f18e","Type":"ContainerDied","Data":"3fb635f53376a131d56fa02140b4d4ea808aec5e003be665e407be8bcd573914"}
Nov 03 14:38:05.255096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.250599    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.284539    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials\") pod \"24dead9f-0c88-40cb-871f-9f27c683f18e\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") "
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.284642    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume\") pod \"24dead9f-0c88-40cb-871f-9f27c683f18e\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") "
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.284759    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr\") pod \"24dead9f-0c88-40cb-871f-9f27c683f18e\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") "
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.284789    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume\") pod \"24dead9f-0c88-40cb-871f-9f27c683f18e\" (UID: \"24dead9f-0c88-40cb-871f-9f27c683f18e\") "
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.285246    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume" pod "24dead9f-0c88-40cb-871f-9f27c683f18e" (UID: "24dead9f-0c88-40cb-871f-9f27c683f18e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.285347    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume" pod "24dead9f-0c88-40cb-871f-9f27c683f18e" (UID: "24dead9f-0c88-40cb-871f-9f27c683f18e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.286038    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:05.286169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.286058    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24dead9f-0c88-40cb-871f-9f27c683f18e-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:05.290636 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.290608    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials" pod "24dead9f-0c88-40cb-871f-9f27c683f18e" (UID: "24dead9f-0c88-40cb-871f-9f27c683f18e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:05.294859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.294817    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr" pod "24dead9f-0c88-40cb-871f-9f27c683f18e" (UID: "24dead9f-0c88-40cb-871f-9f27c683f18e"). InnerVolumeSpecName "kube-api-access-bffxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:05.386878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.386796    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bffxr\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-kube-api-access-bffxr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:05.386878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:05.386827    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24dead9f-0c88-40cb-871f-9f27c683f18e-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:06.068874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.068838    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:06.094418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.094382    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"
Nov 03 14:38:06.094418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.094398    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg" event={"ID":"24dead9f-0c88-40cb-871f-9f27c683f18e","Type":"ContainerDied","Data":"8d52ea2c684b72354ce3737ab13c3dff1f95cf68862f7558fd8e23faad6b6662"}
Nov 03 14:38:06.094616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.094459    6481 scope.go:122] "RemoveContainer" containerID="3fb635f53376a131d56fa02140b4d4ea808aec5e003be665e407be8bcd573914"
Nov 03 14:38:06.153076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.153040    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"]
Nov 03 14:38:06.164496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.164466    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-69ff5qhzg"]
Nov 03 14:38:06.887682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.887649    6481 scope.go:122] "RemoveContainer" containerID="bf2091cb6dadbded5ac03c705ba8f7bd20cbb665b9c0df00fe9770fcd8bf6133"
Nov 03 14:38:06.897590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.897563    6481 scope.go:122] "RemoveContainer" containerID="0c772314efa05fafeb2e46c60a9be4e8900e8c8f9955e95ae2b7c173fcf0f10f"
Nov 03 14:38:06.906291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.906272    6481 scope.go:122] "RemoveContainer" containerID="bff5997da6f2a691cf51f00c9f34321feee923925eaefa93a6b2d97d21c27603"
Nov 03 14:38:06.914785 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:06.914767    6481 scope.go:122] "RemoveContainer" containerID="8fdf54fee0fa4c060a8c2ea4300cf8d6f88f2e7fa6b0bb778bf64a4d6315b928"
Nov 03 14:38:08.040770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:08.040732    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"]
Nov 03 14:38:08.051586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:08.051561    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b75881ae-dgtfk"]
Nov 03 14:38:08.067265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:08.067222    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c8b489-a9c0-4059-b654-11baadb6d4a8" path="/var/lib/kubelet/pods/22c8b489-a9c0-4059-b654-11baadb6d4a8/volumes"
Nov 03 14:38:08.067669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:08.067655    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24dead9f-0c88-40cb-871f-9f27c683f18e" path="/var/lib/kubelet/pods/24dead9f-0c88-40cb-871f-9f27c683f18e/volumes"
Nov 03 14:38:08.248974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:08.248945    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:09.005179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.005145    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:09.214492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.214463    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:09.372397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.372318    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"]
Nov 03 14:38:09.372901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.372886    6481 cpu_manager.go:414] "containerMap: removing container" podUID="445bc7c6-b5d1-4060-94da-77c8ec523f7b" containerName="revoke-keys"
Nov 03 14:38:09.372948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.372903    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445bc7c6-b5d1-4060-94da-77c8ec523f7b" containerName="revoke-keys"
Nov 03 14:38:09.372948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.372922    6481 cpu_manager.go:414] "containerMap: removing container" podUID="24dead9f-0c88-40cb-871f-9f27c683f18e" containerName="payload-processing"
Nov 03 14:38:09.372948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.372928    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24dead9f-0c88-40cb-871f-9f27c683f18e" containerName="payload-processing"
Nov 03 14:38:09.373044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.373035    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="445bc7c6-b5d1-4060-94da-77c8ec523f7b" containerName="revoke-keys"
Nov 03 14:38:09.373080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.373046    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="24dead9f-0c88-40cb-871f-9f27c683f18e" containerName="payload-processing"
Nov 03 14:38:09.378306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.378285    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.409909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.409858    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"]
Nov 03 14:38:09.471252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.466024    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5h7m\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.471252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.466156    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.471252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.466220    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.471252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.466441    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.568162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.568133    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.568364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.568181    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.568364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.568284    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.568364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.568348    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5h7m\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.568918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.568888    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.569027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.568937    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.571516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.571491    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.589179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.589155    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5h7m\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:09.701152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:09.701128    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:10.010019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.009985    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"]
Nov 03 14:38:10.010593 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:10.010565    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d61e94_82df_47b6_a546_93496d1af8e8.slice/crio-36152107fe5cd17cc1fb4b2c6291524b634a015b679d2986b3c528ba9198768d WatchSource:0}: Error finding container 36152107fe5cd17cc1fb4b2c6291524b634a015b679d2986b3c528ba9198768d: Status 404 returned error can't find the container with id 36152107fe5cd17cc1fb4b2c6291524b634a015b679d2986b3c528ba9198768d
Nov 03 14:38:10.130254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.129014    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:10.143547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.141416    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" event={"ID":"31d61e94-82df-47b6-a546-93496d1af8e8","Type":"ContainerStarted","Data":"b578d32a16d9c7ac3114499b8200c566ab2e40ddf260a5e6d1ebe6a9a5a64912"}
Nov 03 14:38:10.143547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.141489    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" event={"ID":"31d61e94-82df-47b6-a546-93496d1af8e8","Type":"ContainerStarted","Data":"36152107fe5cd17cc1fb4b2c6291524b634a015b679d2986b3c528ba9198768d"}
Nov 03 14:38:10.143547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.142568    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:10.176862 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:10.176821    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-90ead7998bc6: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:10.176862 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:10.176854    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:10.177085 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:10.176916    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials podName:31d61e94-82df-47b6-a546-93496d1af8e8 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:10.676895123 +0000 UTC m=+1568.726926309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8") : secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:10.190135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.190071    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" podStartSLOduration=1.190051734 podStartE2EDuration="1.190051734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:10.186984379 +0000 UTC m=+1568.237015562" watchObservedRunningTime="2026-11-03 14:38:10.190051734 +0000 UTC m=+1568.240082918"
Nov 03 14:38:10.367941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.367863    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:10.588465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.588417    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:10.654086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.654007    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"]
Nov 03 14:38:10.658533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.658513    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.674296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.674269    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:38:10.677275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.677219    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"]
Nov 03 14:38:10.684613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.684591    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:38:10.684748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.684617    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:38:10.684748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.684668    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:38:10.685409 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:10.685389    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-90ead7998bc6: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:10.685409 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:10.685411    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:10.685594 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:10.685449    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials podName:31d61e94-82df-47b6-a546-93496d1af8e8 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:11.68543754 +0000 UTC m=+1569.735468714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8") : secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:10.786705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.786674    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.786910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.786837    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mw9\" (UniqueName: \"kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.787034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.787019    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.888254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.888208    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.888414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.888291    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.888414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.888388    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mw9\" (UniqueName: \"kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.888997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.888972    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.890396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.890372    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.909135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.909111    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mw9\" (UniqueName: \"kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9\") pod \"maas-api-revoke-sub-54af72fa-579td\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:10.973022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:10.972998    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:11.274527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.274450    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"]
Nov 03 14:38:11.282070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.281080    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"]
Nov 03 14:38:11.282070 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:11.281292    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9835a0_016e_46e1_a9d1_398b98635eb7.slice/crio-c181b45aa3762b29d2694690970a611890d636ab6c7e888a5f6b327d645c93f2 WatchSource:0}: Error finding container c181b45aa3762b29d2694690970a611890d636ab6c7e888a5f6b327d645c93f2: Status 404 returned error can't find the container with id c181b45aa3762b29d2694690970a611890d636ab6c7e888a5f6b327d645c93f2
Nov 03 14:38:11.310697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.310671    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"]
Nov 03 14:38:11.316024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.316006    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.338139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.338104    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"]
Nov 03 14:38:11.395734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.395696    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.396131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.395826    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.396131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.395900    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.396131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.396063    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz6g\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.497058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.497025    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz6g\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.497215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.497081    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.497215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.497113    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.497215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.497149    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.497818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.497800    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.497953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.497935    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.501162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.501140    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.517988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.517967    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz6g\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.629011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.628985    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:11.702185 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:11.701056    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-90ead7998bc6: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:11.702185 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:11.701093    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:11.702185 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:11.701158    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials podName:31d61e94-82df-47b6-a546-93496d1af8e8 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:13.701139216 +0000 UTC m=+1571.751170397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8") : secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:11.938756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:11.938727    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"]
Nov 03 14:38:11.939406 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:11.939376    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode831edb9_8e4a_4070_91ab_4f1709b64ff7.slice/crio-9088039df70e62cc9bf9fa1f055cc4e6e698571d773de619c07fa486cafd5483 WatchSource:0}: Error finding container 9088039df70e62cc9bf9fa1f055cc4e6e698571d773de619c07fa486cafd5483: Status 404 returned error can't find the container with id 9088039df70e62cc9bf9fa1f055cc4e6e698571d773de619c07fa486cafd5483
Nov 03 14:38:12.038779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.038731    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:12.156595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.156559    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td" event={"ID":"2f9835a0-016e-46e1-a9d1-398b98635eb7","Type":"ContainerStarted","Data":"eb0046464bf537ef6490c06dc2baa176b1faac77fb7377b912d8307c87f7e145"}
Nov 03 14:38:12.156746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.156625    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td" event={"ID":"2f9835a0-016e-46e1-a9d1-398b98635eb7","Type":"ContainerStarted","Data":"c181b45aa3762b29d2694690970a611890d636ab6c7e888a5f6b327d645c93f2"}
Nov 03 14:38:12.158220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.158193    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" event={"ID":"e831edb9-8e4a-4070-91ab-4f1709b64ff7","Type":"ContainerStarted","Data":"55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba"}
Nov 03 14:38:12.158340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.158255    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" event={"ID":"e831edb9-8e4a-4070-91ab-4f1709b64ff7","Type":"ContainerStarted","Data":"9088039df70e62cc9bf9fa1f055cc4e6e698571d773de619c07fa486cafd5483"}
Nov 03 14:38:12.158340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.158283    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:12.224131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.224055    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td" podStartSLOduration=2.224044403 podStartE2EDuration="2.224044403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:12.213534532 +0000 UTC m=+1570.263565715" watchObservedRunningTime="2026-11-03 14:38:12.224044403 +0000 UTC m=+1570.274075633"
Nov 03 14:38:12.274975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.274953    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:12.292508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.292467    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" podStartSLOduration=1.2924538349999999 podStartE2EDuration="1.292453835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:12.281554368 +0000 UTC m=+1570.331585551" watchObservedRunningTime="2026-11-03 14:38:12.292453835 +0000 UTC m=+1570.342485039"
Nov 03 14:38:12.671102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.671078    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:38:12.708769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.708741    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:38:12.717108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.717085    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb\" (UniqueName: \"kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb\") pod \"c900825a-60e4-4b3b-ab2d-a98b10460b97\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") "
Nov 03 14:38:12.717249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.717142    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume\") pod \"c900825a-60e4-4b3b-ab2d-a98b10460b97\" (UID: \"c900825a-60e4-4b3b-ab2d-a98b10460b97\") "
Nov 03 14:38:12.717589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.717561    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume" pod "c900825a-60e4-4b3b-ab2d-a98b10460b97" (UID: "c900825a-60e4-4b3b-ab2d-a98b10460b97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:12.717700 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.717672    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c900825a-60e4-4b3b-ab2d-a98b10460b97-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:12.719510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.719487    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb" pod "c900825a-60e4-4b3b-ab2d-a98b10460b97" (UID: "c900825a-60e4-4b3b-ab2d-a98b10460b97"). InnerVolumeSpecName "kube-api-access-952tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:12.792088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.792008    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:12.818980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.818951    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q\" (UniqueName: \"kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q\") pod \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") "
Nov 03 14:38:12.819248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.819215    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume\") pod \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\" (UID: \"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25\") "
Nov 03 14:38:12.819541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.819517    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume" pod "ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" (UID: "ccd80f7f-50a8-4b3b-8fa2-595dfec95e25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:12.819708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.819690    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:12.819752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.819712    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-952tb\" (UniqueName: \"kubernetes.io/projected/c900825a-60e4-4b3b-ab2d-a98b10460b97-kube-api-access-952tb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:12.821003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.820986    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q" pod "ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" (UID: "ccd80f7f-50a8-4b3b-8fa2-595dfec95e25"). InnerVolumeSpecName "kube-api-access-ctv9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:12.921183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.921145    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctv9q\" (UniqueName: \"kubernetes.io/projected/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25-kube-api-access-ctv9q\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:12.995206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:12.995174    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:13.076726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.076655    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"]
Nov 03 14:38:13.077223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.077211    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerName="payload-processing"
Nov 03 14:38:13.077292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.077225    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerName="payload-processing"
Nov 03 14:38:13.077292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.077271    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerName="payload-pre-processing"
Nov 03 14:38:13.077292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.077277    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerName="payload-pre-processing"
Nov 03 14:38:13.077480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.077360    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerName="payload-processing"
Nov 03 14:38:13.077480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.077371    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerName="payload-pre-processing"
Nov 03 14:38:13.081094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.081074    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.093340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.093315    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"]
Nov 03 14:38:13.098386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.098368    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"]
Nov 03 14:38:13.098480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.098467    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.103422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.102717    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.106587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.106564    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"]
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.123745    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.123802    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.123830    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tnn2\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.123902    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.123947    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.123977    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.124009    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.124069    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.124139    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.128159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.127256    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"]
Nov 03 14:38:13.135391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.135352    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"]
Nov 03 14:38:13.137752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.137728    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\""
Nov 03 14:38:13.168951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.168923    6481 generic.go:356] "Generic (PLEG): container finished" podID="2f9835a0-016e-46e1-a9d1-398b98635eb7" containerID="eb0046464bf537ef6490c06dc2baa176b1faac77fb7377b912d8307c87f7e145" exitCode=0
Nov 03 14:38:13.169078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.169009    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td" event={"ID":"2f9835a0-016e-46e1-a9d1-398b98635eb7","Type":"ContainerDied","Data":"eb0046464bf537ef6490c06dc2baa176b1faac77fb7377b912d8307c87f7e145"}
Nov 03 14:38:13.170437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.170415    6481 generic.go:356] "Generic (PLEG): container finished" podID="c900825a-60e4-4b3b-ab2d-a98b10460b97" containerID="2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8" exitCode=0
Nov 03 14:38:13.170533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.170481    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"
Nov 03 14:38:13.170533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.170499    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" event={"ID":"c900825a-60e4-4b3b-ab2d-a98b10460b97","Type":"ContainerDied","Data":"2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8"}
Nov 03 14:38:13.170646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.170543    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22" event={"ID":"c900825a-60e4-4b3b-ab2d-a98b10460b97","Type":"ContainerDied","Data":"950efcd5721b8879814b286472c9b3274ae16d2dd7a0350d185d4883f43de986"}
Nov 03 14:38:13.170646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.170567    6481 scope.go:122] "RemoveContainer" containerID="2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8"
Nov 03 14:38:13.172065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.172045    6481 generic.go:356] "Generic (PLEG): container finished" podID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" containerID="dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2" exitCode=0
Nov 03 14:38:13.172158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.172101    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"
Nov 03 14:38:13.172158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.172113    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" event={"ID":"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25","Type":"ContainerDied","Data":"dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2"}
Nov 03 14:38:13.172287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.172160    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl" event={"ID":"ccd80f7f-50a8-4b3b-8fa2-595dfec95e25","Type":"ContainerDied","Data":"3ce8cb8a1691904e3f01b3b0a70ff7643c3bae3ec2361a12ce5ef11e8189b77e"}
Nov 03 14:38:13.183412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.183384    6481 scope.go:122] "RemoveContainer" containerID="2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8"
Nov 03 14:38:13.183761 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:13.183731    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8\": container with ID starting with 2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8 not found: ID does not exist" containerID="2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8"
Nov 03 14:38:13.183851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.183768    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8"} err="failed to get container status \"2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8\": rpc error: code = NotFound desc = could not find container \"2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8\": container with ID starting with 2b0839ffc32656a31ffa30095c10686069e11af7e3ccedc02055b6d1d0e718b8 not found: ID does not exist"
Nov 03 14:38:13.183851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.183785    6481 scope.go:122] "RemoveContainer" containerID="dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2"
Nov 03 14:38:13.195496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.195030    6481 scope.go:122] "RemoveContainer" containerID="dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2"
Nov 03 14:38:13.195496 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:13.195372    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2\": container with ID starting with dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2 not found: ID does not exist" containerID="dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2"
Nov 03 14:38:13.195496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.195409    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2"} err="failed to get container status \"dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2\": rpc error: code = NotFound desc = could not find container \"dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2\": container with ID starting with dd550911016fd1084d0f4c1142d8208b06e557d25156c015e2c366e6dbb199d2 not found: ID does not exist"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225588    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225639    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225761    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225806    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225861    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225923    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5685\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225953    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.225982    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.226372    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.226519    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.226550    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.226578    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.226639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.226605    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj46r\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227107    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227141    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tnn2\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227190    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227278    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227307    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227355    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227397    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227436    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227481    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227530    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227690    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.227965    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.228143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228060    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.229778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228299    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.229778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228374    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.229778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228399    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.229778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228435    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.229778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228473    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.229778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.228626    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.232919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.232874    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.239757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.239736    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"]
Nov 03 14:38:13.241265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.241225    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy" containerID="cri-o://d46616747896d7f41d4a0b2d196adb6cb16cb451e0c6c11018e01ea2197b2927" gracePeriod=30
Nov 03 14:38:13.246341 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.246296    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"]
Nov 03 14:38:13.246620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.246501    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy" containerID="cri-o://6ba5888ab83638817c34d40b1ec860af341203d4d2be76578aa2f1df95b1bcd1" gracePeriod=30
Nov 03 14:38:13.248668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.248488    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tnn2\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.249242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.249210    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.255096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.255072    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.290048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.290021    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"]
Nov 03 14:38:13.296268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.296248    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-86c2e2-7b559c4579-j6z22"]
Nov 03 14:38:13.340632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340532    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.340632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340608    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.340886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340660    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.340886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340712    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.340886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340742    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.340886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340807    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.340886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340831    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.340886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340868    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340912    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.340942    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341008    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341081    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341134    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5685\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341165    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.341211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341196    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.349676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341287    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.349676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341314    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.349676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.341346    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj46r\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.349676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.342100    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.349676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.342384    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.349676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.342719    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.350767    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.351055    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.352378    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.352924    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.353160    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.353412    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.354795    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.354990    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.358142    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.358983    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"]
Nov 03 14:38:13.373253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.361344    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-86c2e2-d85d5ffc5-wcgpl"]
Nov 03 14:38:13.379777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.379691    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj46r\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.381845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.381816    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.383585 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.383559    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5685\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.387557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.387509    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.388065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.388043    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.389327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.389177    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.390821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.390796    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo\") pod \"e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") " pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.398374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.398219    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:13.422818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.422765    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:13.437382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.437350    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:13.745153 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:13.745119    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bdaf4e_ad15_419f_9c68_126fecb418fe.slice/crio-53192d52c772d3a10bfcd34a193e78cebd98d3fe6bd77713837cf55a9b075133 WatchSource:0}: Error finding container 53192d52c772d3a10bfcd34a193e78cebd98d3fe6bd77713837cf55a9b075133: Status 404 returned error can't find the container with id 53192d52c772d3a10bfcd34a193e78cebd98d3fe6bd77713837cf55a9b075133
Nov 03 14:38:13.746274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.746216    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"]
Nov 03 14:38:13.748112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.748061    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.748219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.748151    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.748219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.748207    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.749347 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:13.749330    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-90ead7998bc6: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:13.749429 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:13.749351    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:13.749429 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:13.749410    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials podName:31d61e94-82df-47b6-a546-93496d1af8e8 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:17.74939012 +0000 UTC m=+1575.799421296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8") : secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:13.761126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.761098    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"]
Nov 03 14:38:13.762313 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:13.762288    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c804f17_3e91_41ff_99f5_5a2ad073f3ed.slice/crio-63d5a7581555052030dfa106396326df5f9083b04c9db806738d76694e1ea065 WatchSource:0}: Error finding container 63d5a7581555052030dfa106396326df5f9083b04c9db806738d76694e1ea065: Status 404 returned error can't find the container with id 63d5a7581555052030dfa106396326df5f9083b04c9db806738d76694e1ea065
Nov 03 14:38:13.764825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.764790    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.764918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.764856    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.764918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.764891    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.788760 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:13.788735    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90630806_857f_41ea_8571_f0c8439c1fbe.slice/crio-b2cd4717f1a27558347a62fd7ecc4f30bbcdc701a190763309e391f87bc90722 WatchSource:0}: Error finding container b2cd4717f1a27558347a62fd7ecc4f30bbcdc701a190763309e391f87bc90722: Status 404 returned error can't find the container with id b2cd4717f1a27558347a62fd7ecc4f30bbcdc701a190763309e391f87bc90722
Nov 03 14:38:13.790728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.790706    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"]
Nov 03 14:38:13.791031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.791003    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.791113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.791073    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:13.791171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:13.791123    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:14.084803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.084773    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:14.088309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.088278    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c900825a-60e4-4b3b-ab2d-a98b10460b97" path="/var/lib/kubelet/pods/c900825a-60e4-4b3b-ab2d-a98b10460b97/volumes"
Nov 03 14:38:14.088928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.088905    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccd80f7f-50a8-4b3b-8fa2-595dfec95e25" path="/var/lib/kubelet/pods/ccd80f7f-50a8-4b3b-8fa2-595dfec95e25/volumes"
Nov 03 14:38:14.184809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.184768    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" event={"ID":"20bdaf4e-ad15-419f-9c68-126fecb418fe","Type":"ContainerStarted","Data":"a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f"}
Nov 03 14:38:14.185015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.184839    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" event={"ID":"20bdaf4e-ad15-419f-9c68-126fecb418fe","Type":"ContainerStarted","Data":"53192d52c772d3a10bfcd34a193e78cebd98d3fe6bd77713837cf55a9b075133"}
Nov 03 14:38:14.186690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.186665    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" event={"ID":"0c804f17-3e91-41ff-99f5-5a2ad073f3ed","Type":"ContainerStarted","Data":"374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a"}
Nov 03 14:38:14.186822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.186727    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" event={"ID":"0c804f17-3e91-41ff-99f5-5a2ad073f3ed","Type":"ContainerStarted","Data":"63d5a7581555052030dfa106396326df5f9083b04c9db806738d76694e1ea065"}
Nov 03 14:38:14.189022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.188992    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" event={"ID":"90630806-857f-41ea-8571-f0c8439c1fbe","Type":"ContainerStarted","Data":"62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5"}
Nov 03 14:38:14.189165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.189043    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" event={"ID":"90630806-857f-41ea-8571-f0c8439c1fbe","Type":"ContainerStarted","Data":"b2cd4717f1a27558347a62fd7ecc4f30bbcdc701a190763309e391f87bc90722"}
Nov 03 14:38:14.236656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.236593    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podStartSLOduration=1.236575869 podStartE2EDuration="1.236575869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:14.230299826 +0000 UTC m=+1572.280331009" watchObservedRunningTime="2026-11-03 14:38:14.236575869 +0000 UTC m=+1572.286607050"
Nov 03 14:38:14.359084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.353321    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podStartSLOduration=1.353296888 podStartE2EDuration="1.353296888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:14.300765403 +0000 UTC m=+1572.350796587" watchObservedRunningTime="2026-11-03 14:38:14.353296888 +0000 UTC m=+1572.403328080"
Nov 03 14:38:14.410892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.402623    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:14.424403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.421490    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:14.424944 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.421581    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:14.425183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.425154    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:14.457542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.457506    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:14.463414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.463382    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:14.463641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.458338    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:14.463779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.463758    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:14.485318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.485221    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podStartSLOduration=1.485203349 podStartE2EDuration="1.485203349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:14.366102816 +0000 UTC m=+1572.416133994" watchObservedRunningTime="2026-11-03 14:38:14.485203349 +0000 UTC m=+1572.535234534"
Nov 03 14:38:14.486446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.486418    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:14.486657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.486633    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:14.591084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.591054    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca\") pod \"2f9835a0-016e-46e1-a9d1-398b98635eb7\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") "
Nov 03 14:38:14.591198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.591179    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9\" (UniqueName: \"kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9\") pod \"2f9835a0-016e-46e1-a9d1-398b98635eb7\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") "
Nov 03 14:38:14.591315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.591295    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca\") pod \"2f9835a0-016e-46e1-a9d1-398b98635eb7\" (UID: \"2f9835a0-016e-46e1-a9d1-398b98635eb7\") "
Nov 03 14:38:14.591846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.591820    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca" pod "2f9835a0-016e-46e1-a9d1-398b98635eb7" (UID: "2f9835a0-016e-46e1-a9d1-398b98635eb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:14.592093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.592074    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:14.593611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.593583    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9" pod "2f9835a0-016e-46e1-a9d1-398b98635eb7" (UID: "2f9835a0-016e-46e1-a9d1-398b98635eb7"). InnerVolumeSpecName "kube-api-access-t6mw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:14.594066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.594043    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca" pod "2f9835a0-016e-46e1-a9d1-398b98635eb7" (UID: "2f9835a0-016e-46e1-a9d1-398b98635eb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:38:14.693985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.693955    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f9835a0-016e-46e1-a9d1-398b98635eb7-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:14.693985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:14.693983    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mw9\" (UniqueName: \"kubernetes.io/projected/2f9835a0-016e-46e1-a9d1-398b98635eb7-kube-api-access-t6mw9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:15.081440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.081403    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:15.290387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.290344    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td" event={"ID":"2f9835a0-016e-46e1-a9d1-398b98635eb7","Type":"ContainerDied","Data":"c181b45aa3762b29d2694690970a611890d636ab6c7e888a5f6b327d645c93f2"}
Nov 03 14:38:15.290638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.290623    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c181b45aa3762b29d2694690970a611890d636ab6c7e888a5f6b327d645c93f2"
Nov 03 14:38:15.290842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.290816    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"
Nov 03 14:38:15.399202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.399088    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:15.399202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.399148    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:15.444255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.443915    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:15.444255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.443972    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:15.444545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.444304    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:15.444545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:15.444356    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:16.399766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.399727    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:16.400300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.399791    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:16.436747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.436622    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:16.436747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.436685    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:16.441108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.440536    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:16.441108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.440581    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:16.765484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.765449    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:16.986445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:16.986415    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:17.504107 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.504068    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:17.566254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.544243    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:17.623252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.541558    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:17.623673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.623630    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:17.624263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.623348    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:17.663026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.655601    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:17.663400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.662990    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:17.823306 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:17.822729    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-90ead7998bc6: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:17.823306 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:17.822766    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb: secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:17.823306 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:17.822854    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials podName:31d61e94-82df-47b6-a546-93496d1af8e8 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:25.82283464 +0000 UTC m=+1583.872865815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8") : secret "agc-cred-90ead7998bc6" not found
Nov 03 14:38:17.877901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:17.877850    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:18.107295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.107192    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:18.177357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.177306    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" podUID="31d61e94-82df-47b6-a546-93496d1af8e8" containerName="payload-processing" containerID="cri-o://b578d32a16d9c7ac3114499b8200c566ab2e40ddf260a5e6d1ebe6a9a5a64912" gracePeriod=25
Nov 03 14:38:18.420119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.418605    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:18.420119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.418658    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:18.428412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.427336    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:18.428412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.427405    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:18.454674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.440588    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:18.454674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.440636    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:18.798462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.796431    6481 generic.go:356] "Generic (PLEG): container finished" podID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerID="d46616747896d7f41d4a0b2d196adb6cb16cb451e0c6c11018e01ea2197b2927" exitCode=0
Nov 03 14:38:18.798462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.796542    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" event={"ID":"506204d2-8aa7-4d8c-9c70-4abfc29836c6","Type":"ContainerDied","Data":"d46616747896d7f41d4a0b2d196adb6cb16cb451e0c6c11018e01ea2197b2927"}
Nov 03 14:38:18.798462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.798193    6481 generic.go:356] "Generic (PLEG): container finished" podID="31d61e94-82df-47b6-a546-93496d1af8e8" containerID="b578d32a16d9c7ac3114499b8200c566ab2e40ddf260a5e6d1ebe6a9a5a64912" exitCode=0
Nov 03 14:38:18.798462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.798285    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" event={"ID":"31d61e94-82df-47b6-a546-93496d1af8e8","Type":"ContainerDied","Data":"b578d32a16d9c7ac3114499b8200c566ab2e40ddf260a5e6d1ebe6a9a5a64912"}
Nov 03 14:38:18.819839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.800020    6481 generic.go:356] "Generic (PLEG): container finished" podID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerID="6ba5888ab83638817c34d40b1ec860af341203d4d2be76578aa2f1df95b1bcd1" exitCode=0
Nov 03 14:38:18.819839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.800147    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" event={"ID":"9afb21a5-fe6f-410a-949b-41ef42ee1ccb","Type":"ContainerDied","Data":"6ba5888ab83638817c34d40b1ec860af341203d4d2be76578aa2f1df95b1bcd1"}
Nov 03 14:38:18.992442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:18.992408    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:38:19.193749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.193695    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.193932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.193810    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.193932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.193907    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.193960    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194036    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194096    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194205    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194297    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194333    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert\") pod \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\" (UID: \"506204d2-8aa7-4d8c-9c70-4abfc29836c6\") "
Nov 03 14:38:19.194699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194671    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.194975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.194956    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.195155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.195139    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.195265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.195081    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:19.195353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.195335    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.197076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.197041    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.198661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.198635    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "kube-api-access-25lfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:19.199596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.199571    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:19.230263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.208326    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:19.230263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.218266    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy" pod "506204d2-8aa7-4d8c-9c70-4abfc29836c6" (UID: "506204d2-8aa7-4d8c-9c70-4abfc29836c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298330    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298385    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298395    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298405    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298413    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298422    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298447    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25lfv\" (UniqueName: \"kubernetes.io/projected/506204d2-8aa7-4d8c-9c70-4abfc29836c6-kube-api-access-25lfv\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.299252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.298456    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506204d2-8aa7-4d8c-9c70-4abfc29836c6-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.429255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.398990    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:19.429255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.399052    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:19.430525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.430409    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:19.430525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.430467    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:19.463463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.463289    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:19.463463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.463359    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:19.600092 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.600023    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:38:19.652427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.652393    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.762648    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.762725    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.762792    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.762883    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.762920    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.763031    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.763062    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.763119    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.767307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.763248    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket\") pod \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\" (UID: \"9afb21a5-fe6f-410a-949b-41ef42ee1ccb\") "
Nov 03 14:38:19.775490 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.775124    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:19.777045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.776775    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.777508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.777222    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.777878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.777847    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.779575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.779550    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.781311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.781268    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "kube-api-access-sn9rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:19.782640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.782405    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:19.792268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.792216    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:19.792376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.792293    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token" pod "9afb21a5-fe6f-410a-949b-41ef42ee1ccb" (UID: "9afb21a5-fe6f-410a-949b-41ef42ee1ccb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.877742    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials\") pod \"31d61e94-82df-47b6-a546-93496d1af8e8\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") "
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.877810    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m\") pod \"31d61e94-82df-47b6-a546-93496d1af8e8\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") "
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.877968    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume\") pod \"31d61e94-82df-47b6-a546-93496d1af8e8\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") "
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878080    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume\") pod \"31d61e94-82df-47b6-a546-93496d1af8e8\" (UID: \"31d61e94-82df-47b6-a546-93496d1af8e8\") "
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878658    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878682    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878696    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878709    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878723    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878736    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878750    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn9rk\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-kube-api-access-sn9rk\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878764    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.881327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.878782    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9afb21a5-fe6f-410a-949b-41ef42ee1ccb-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.883971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.883598    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume" pod "31d61e94-82df-47b6-a546-93496d1af8e8" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:19.884304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.884267    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume" pod "31d61e94-82df-47b6-a546-93496d1af8e8" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:19.889283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.889216    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6" event={"ID":"506204d2-8aa7-4d8c-9c70-4abfc29836c6","Type":"ContainerDied","Data":"29d8cf5fb0f0a6d2827eda2f988cae504303f36b13451112880f8ad424e047d0"}
Nov 03 14:38:19.889421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.889323    6481 scope.go:122] "RemoveContainer" containerID="d46616747896d7f41d4a0b2d196adb6cb16cb451e0c6c11018e01ea2197b2927"
Nov 03 14:38:19.889421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.889374    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m" pod "31d61e94-82df-47b6-a546-93496d1af8e8" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8"). InnerVolumeSpecName "kube-api-access-z5h7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:19.889542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.889519    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"
Nov 03 14:38:19.891849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.891648    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials" pod "31d61e94-82df-47b6-a546-93496d1af8e8" (UID: "31d61e94-82df-47b6-a546-93496d1af8e8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:19.892501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.892477    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"
Nov 03 14:38:19.892622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.892585    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb" event={"ID":"31d61e94-82df-47b6-a546-93496d1af8e8","Type":"ContainerDied","Data":"36152107fe5cd17cc1fb4b2c6291524b634a015b679d2986b3c528ba9198768d"}
Nov 03 14:38:19.894964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.894626    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"
Nov 03 14:38:19.894964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.894632    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77" event={"ID":"9afb21a5-fe6f-410a-949b-41ef42ee1ccb","Type":"ContainerDied","Data":"7814c2c4b92434322f8dd44f64604270de5818a731ce546f39befe46c41f6dcd"}
Nov 03 14:38:19.986298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.985471    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.986298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.985502    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31d61e94-82df-47b6-a546-93496d1af8e8-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.986298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.985516    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.986298 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.985525    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5h7m\" (UniqueName: \"kubernetes.io/projected/31d61e94-82df-47b6-a546-93496d1af8e8-kube-api-access-z5h7m\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:19.987981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:19.987959    6481 scope.go:122] "RemoveContainer" containerID="b578d32a16d9c7ac3114499b8200c566ab2e40ddf260a5e6d1ebe6a9a5a64912"
Nov 03 14:38:20.044351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.044301    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"]
Nov 03 14:38:20.087629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.086433    6481 scope.go:122] "RemoveContainer" containerID="6ba5888ab83638817c34d40b1ec860af341203d4d2be76578aa2f1df95b1bcd1"
Nov 03 14:38:20.096906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.096071    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-jvqw6"]
Nov 03 14:38:20.096906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.096107    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"]
Nov 03 14:38:20.110256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.099464    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7694k5wnb"]
Nov 03 14:38:20.188842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.188789    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"]
Nov 03 14:38:20.192316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.192291    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-1ffaa0-openshift-default-58876d79fc-45v77"]
Nov 03 14:38:20.400203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.400158    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:20.401029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.400994    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:20.433501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.433463    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:20.433686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.433527    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:20.439088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.439056    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:20.439338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.439307    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:20.452883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:20.452840    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:21.042641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.042599    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"]
Nov 03 14:38:21.050388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.050357    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd49d7ea-7vc8b"]
Nov 03 14:38:21.399225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.399135    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Nov 03 14:38:21.399225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.399201    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Nov 03 14:38:21.423558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.423522    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Nov 03 14:38:21.423696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.423581    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Nov 03 14:38:21.437656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.437631    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Nov 03 14:38:21.437756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:21.437682    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Nov 03 14:38:22.072804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:22.072769    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d61e94-82df-47b6-a546-93496d1af8e8" path="/var/lib/kubelet/pods/31d61e94-82df-47b6-a546-93496d1af8e8/volumes"
Nov 03 14:38:22.074210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:22.074186    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" path="/var/lib/kubelet/pods/506204d2-8aa7-4d8c-9c70-4abfc29836c6/volumes"
Nov 03 14:38:22.075010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:22.074989    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fe30dc5-d54b-40e7-a787-65276b05329a" path="/var/lib/kubelet/pods/7fe30dc5-d54b-40e7-a787-65276b05329a/volumes"
Nov 03 14:38:22.076212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:22.076193    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" path="/var/lib/kubelet/pods/9afb21a5-fe6f-410a-949b-41ef42ee1ccb/volumes"
Nov 03 14:38:22.608275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:22.604661    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:23.119445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.119409    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:23.203151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.203110    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:23.249184 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:23.249151    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-386f6d33da21: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:23.377457 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:23.377379    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:23.377716 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:23.377689    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials podName:e831edb9-8e4a-4070-91ab-4f1709b64ff7 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:23.877664132 +0000 UTC m=+1581.927695308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7") : secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:23.377834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.345460    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:23.379997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.379396    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"]
Nov 03 14:38:23.448874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.448809    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:23.460352 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.460305    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:23.486895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.486329    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:23.487156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.487123    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:23.540492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.540442    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:23.540789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.540735    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:23.656486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.656400    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"]
Nov 03 14:38:23.669815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.667670    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"]
Nov 03 14:38:23.688463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.688416    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:23.899504 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:23.899469    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-386f6d33da21: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:23.899749 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:23.899733    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:23.906258 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:23.899891    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials podName:e831edb9-8e4a-4070-91ab-4f1709b64ff7 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:24.899870408 +0000 UTC m=+1582.949901587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7") : secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:23.937258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.933364    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:23.992409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.992368    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"]
Nov 03 14:38:23.993766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.993729    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy"
Nov 03 14:38:23.993961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.993947    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy"
Nov 03 14:38:23.994086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994068    6481 cpu_manager.go:414] "containerMap: removing container" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy"
Nov 03 14:38:23.994173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994163    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy"
Nov 03 14:38:23.994304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994292    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2f9835a0-016e-46e1-a9d1-398b98635eb7" containerName="revoke-keys"
Nov 03 14:38:23.994392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994382    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f9835a0-016e-46e1-a9d1-398b98635eb7" containerName="revoke-keys"
Nov 03 14:38:23.994528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994517    6481 cpu_manager.go:414] "containerMap: removing container" podUID="31d61e94-82df-47b6-a546-93496d1af8e8" containerName="payload-processing"
Nov 03 14:38:23.994628 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994609    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d61e94-82df-47b6-a546-93496d1af8e8" containerName="payload-processing"
Nov 03 14:38:23.994882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994857    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afb21a5-fe6f-410a-949b-41ef42ee1ccb" containerName="istio-proxy"
Nov 03 14:38:23.994990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.994979    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="506204d2-8aa7-4d8c-9c70-4abfc29836c6" containerName="istio-proxy"
Nov 03 14:38:23.995081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.995071    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f9835a0-016e-46e1-a9d1-398b98635eb7" containerName="revoke-keys"
Nov 03 14:38:23.995176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:23.995167    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d61e94-82df-47b6-a546-93496d1af8e8" containerName="payload-processing"
Nov 03 14:38:24.091438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.089371    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.114665    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.115390    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.115497    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.115603    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.117140    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdllr\" (UniqueName: \"kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.117244    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.171688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.117346    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.233311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.218198    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdllr\" (UniqueName: \"kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.282262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.282212    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.282631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.282608    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.294481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.283733    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.336445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.297177    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"]
Nov 03 14:38:24.337321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.337294    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.371370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.371301    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdllr\" (UniqueName: \"kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr\") pod \"maas-api-revoke-sub-5cf274bc-6ct5g\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.499830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.471493    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:24.586776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.586314    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:24.586776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.586372    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:24.592337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.592303    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:24.592457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.592376    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:24.596312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.596277    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:24.620353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:24.620318    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:24.961334 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:24.961296    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-386f6d33da21: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:24.961334 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:24.961332    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:24.961578 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:24.961407    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials podName:e831edb9-8e4a-4070-91ab-4f1709b64ff7 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:26.961384716 +0000 UTC m=+1585.011415901 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7") : secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:25.017865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.017826    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:25.018838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.018809    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:25.035559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.035530    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:25.044863 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:25.044839    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ec9411_5fb7_4467_afc8_e646daa8305c.slice/crio-2e014308c43c67ca5873a9a08bd7734cc06e34d36a90871a04a9d336fe91d775 WatchSource:0}: Error finding container 2e014308c43c67ca5873a9a08bd7734cc06e34d36a90871a04a9d336fe91d775: Status 404 returned error can't find the container with id 2e014308c43c67ca5873a9a08bd7734cc06e34d36a90871a04a9d336fe91d775
Nov 03 14:38:25.046107 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.046076    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"]
Nov 03 14:38:25.087762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.087737    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:25.317317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.306414    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:25.631417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.631327    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:25.631667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.631635    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:25.674371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.674333    6481 patch_prober.go:29] interesting pod/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:38:25.674600 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.674572    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:38:25.888131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.888035    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:25.933086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.933055    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:25.934746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.934723    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:25.938248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:25.938062    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:26.005028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.004992    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:26.006590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.006432    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:26.016830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.016803    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g" event={"ID":"f1ec9411-5fb7-4467-afc8-e646daa8305c","Type":"ContainerStarted","Data":"2ac1734fe87d555818acc14c4a92da398ac7a1fdce0b186ec38ca37c1cf3aa33"}
Nov 03 14:38:26.016964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.016850    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g" event={"ID":"f1ec9411-5fb7-4467-afc8-e646daa8305c","Type":"ContainerStarted","Data":"2e014308c43c67ca5873a9a08bd7734cc06e34d36a90871a04a9d336fe91d775"}
Nov 03 14:38:26.025391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.025368    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:26.160302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.152164    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g" podStartSLOduration=3.152143107 podStartE2EDuration="3.152143107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:26.147618669 +0000 UTC m=+1584.197649867" watchObservedRunningTime="2026-11-03 14:38:26.152143107 +0000 UTC m=+1584.202174293"
Nov 03 14:38:26.180361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.180330    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"]
Nov 03 14:38:26.190740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.190700    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"]
Nov 03 14:38:26.190885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.190767    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"]
Nov 03 14:38:26.258220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.258184    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:26.491758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.491681    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:26.710715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.710678    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:26.977286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:26.977250    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:38:27.011541 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:27.011501    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-386f6d33da21: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:27.011541 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:27.011535    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:27.011780 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:27.011608    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials podName:e831edb9-8e4a-4070-91ab-4f1709b64ff7 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:31.011585814 +0000 UTC m=+1589.061616985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7") : secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:27.025465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:27.025438    6481 generic.go:356] "Generic (PLEG): container finished" podID="f1ec9411-5fb7-4467-afc8-e646daa8305c" containerID="2ac1734fe87d555818acc14c4a92da398ac7a1fdce0b186ec38ca37c1cf3aa33" exitCode=0
Nov 03 14:38:27.025624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:27.025521    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g" event={"ID":"f1ec9411-5fb7-4467-afc8-e646daa8305c","Type":"ContainerDied","Data":"2ac1734fe87d555818acc14c4a92da398ac7a1fdce0b186ec38ca37c1cf3aa33"}
Nov 03 14:38:27.025920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:27.025900    6481 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" secret="" err="secret \"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\" not found"
Nov 03 14:38:27.026065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:27.026048    6481 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" secret="" err="secret \"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\" not found"
Nov 03 14:38:27.026201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:27.026146    6481 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" secret="" err="secret \"e2e-worker-w0-86c2e2-openshift-default-dockercfg-7dv7n\" not found"
Nov 03 14:38:27.468410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:27.468371    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:27.986715 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:27.986681    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:38:28.033079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.033039    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy" containerID="cri-o://62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5" gracePeriod=30
Nov 03 14:38:28.033079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.033063    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy" containerID="cri-o://374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a" gracePeriod=30
Nov 03 14:38:28.033358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.033221    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy" containerID="cri-o://a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f" gracePeriod=30
Nov 03 14:38:28.187945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.187917    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:28.227920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.227892    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr\" (UniqueName: \"kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr\") pod \"f1ec9411-5fb7-4467-afc8-e646daa8305c\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") "
Nov 03 14:38:28.228271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.228026    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca\") pod \"f1ec9411-5fb7-4467-afc8-e646daa8305c\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") "
Nov 03 14:38:28.229278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.228710    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca\") pod \"f1ec9411-5fb7-4467-afc8-e646daa8305c\" (UID: \"f1ec9411-5fb7-4467-afc8-e646daa8305c\") "
Nov 03 14:38:28.229278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.229125    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca" pod "f1ec9411-5fb7-4467-afc8-e646daa8305c" (UID: "f1ec9411-5fb7-4467-afc8-e646daa8305c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:28.229549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.229519    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:28.231534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.231507    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr" pod "f1ec9411-5fb7-4467-afc8-e646daa8305c" (UID: "f1ec9411-5fb7-4467-afc8-e646daa8305c"). InnerVolumeSpecName "kube-api-access-qdllr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:28.231713 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.231695    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca" pod "f1ec9411-5fb7-4467-afc8-e646daa8305c" (UID: "f1ec9411-5fb7-4467-afc8-e646daa8305c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:38:28.330539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.330506    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ec9411-5fb7-4467-afc8-e646daa8305c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:28.330539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.330536    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdllr\" (UniqueName: \"kubernetes.io/projected/f1ec9411-5fb7-4467-afc8-e646daa8305c-kube-api-access-qdllr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:28.380743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.380689    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w" podUID="7f62a75a-1223-4f40-bccc-061e8000e822" containerName="payload-processing" containerID="cri-o://dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd" gracePeriod=25
Nov 03 14:38:28.639823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.639796    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:38:28.657764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.657738    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerName="payload-processing" containerID="cri-o://e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f" gracePeriod=25
Nov 03 14:38:28.669627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.669599    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerName="payload-pre-processing" containerID="cri-o://5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407" gracePeriod=25
Nov 03 14:38:28.733848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.733815    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume\") pod \"7f62a75a-1223-4f40-bccc-061e8000e822\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") "
Nov 03 14:38:28.734009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.733911    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials\") pod \"7f62a75a-1223-4f40-bccc-061e8000e822\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") "
Nov 03 14:38:28.734060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.734045    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf\") pod \"7f62a75a-1223-4f40-bccc-061e8000e822\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") "
Nov 03 14:38:28.734144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.734129    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume\") pod \"7f62a75a-1223-4f40-bccc-061e8000e822\" (UID: \"7f62a75a-1223-4f40-bccc-061e8000e822\") "
Nov 03 14:38:28.734210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.734183    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume" pod "7f62a75a-1223-4f40-bccc-061e8000e822" (UID: "7f62a75a-1223-4f40-bccc-061e8000e822"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:28.734591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.734568    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume" pod "7f62a75a-1223-4f40-bccc-061e8000e822" (UID: "7f62a75a-1223-4f40-bccc-061e8000e822"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:28.734695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.734629    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:28.735956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.735935    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials" pod "7f62a75a-1223-4f40-bccc-061e8000e822" (UID: "7f62a75a-1223-4f40-bccc-061e8000e822"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:28.736036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.735988    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf" pod "7f62a75a-1223-4f40-bccc-061e8000e822" (UID: "7f62a75a-1223-4f40-bccc-061e8000e822"). InnerVolumeSpecName "kube-api-access-phbsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:28.836301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.836198    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phbsf\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-kube-api-access-phbsf\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:28.836301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.836256    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f62a75a-1223-4f40-bccc-061e8000e822-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:28.836301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:28.836268    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f62a75a-1223-4f40-bccc-061e8000e822-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:29.041442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.041410    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"
Nov 03 14:38:29.041824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.041443    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g" event={"ID":"f1ec9411-5fb7-4467-afc8-e646daa8305c","Type":"ContainerDied","Data":"2e014308c43c67ca5873a9a08bd7734cc06e34d36a90871a04a9d336fe91d775"}
Nov 03 14:38:29.041824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.041520    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e014308c43c67ca5873a9a08bd7734cc06e34d36a90871a04a9d336fe91d775"
Nov 03 14:38:29.042878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.042852    6481 generic.go:356] "Generic (PLEG): container finished" podID="7f62a75a-1223-4f40-bccc-061e8000e822" containerID="dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd" exitCode=0
Nov 03 14:38:29.043012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.042888    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"
Nov 03 14:38:29.043012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.042933    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w" event={"ID":"7f62a75a-1223-4f40-bccc-061e8000e822","Type":"ContainerDied","Data":"dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd"}
Nov 03 14:38:29.043012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.042975    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w" event={"ID":"7f62a75a-1223-4f40-bccc-061e8000e822","Type":"ContainerDied","Data":"9be1eb82271bff1713584eee370384afe9f2042a197152d100a2f2380f156497"}
Nov 03 14:38:29.043012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.042996    6481 scope.go:122] "RemoveContainer" containerID="dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd"
Nov 03 14:38:29.056972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.056957    6481 scope.go:122] "RemoveContainer" containerID="dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd"
Nov 03 14:38:29.057288 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:29.057269    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd\": container with ID starting with dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd not found: ID does not exist" containerID="dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd"
Nov 03 14:38:29.057329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.057297    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd"} err="failed to get container status \"dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd\": rpc error: code = NotFound desc = could not find container \"dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd\": container with ID starting with dff431cb368820552291af534562b2a7e0f1bc1bb7b491ea0dd0bea90b788fcd not found: ID does not exist"
Nov 03 14:38:29.096593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.096538    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"]
Nov 03 14:38:29.107606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.107584    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fd8fsf9w"]
Nov 03 14:38:29.499556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:29.499525    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:30.066935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:30.066906    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f62a75a-1223-4f40-bccc-061e8000e822" path="/var/lib/kubelet/pods/7f62a75a-1223-4f40-bccc-061e8000e822/volumes"
Nov 03 14:38:31.067247 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:31.067204    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-386f6d33da21: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:31.067247 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:31.067254    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:31.067764 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:31.067328    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials podName:e831edb9-8e4a-4070-91ab-4f1709b64ff7 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:39.067309298 +0000 UTC m=+1597.117340473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7") : secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:31.135398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:31.135373    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:31.597773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:31.597739    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:32.270862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:32.270826    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:38:32.271348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:32.270826    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:38:33.317704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.317634    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:33.360778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.360755    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:33.398208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.398188    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:33.401280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.401263    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:33.405584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405558    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.405689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405674    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.405758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405709    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.405861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405847    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.405931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405889    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.405985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405933    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.405985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.405931    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:33.406148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406006    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.406148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406032    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.406148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406039    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.406386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406189    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data\") pod \"20bdaf4e-ad15-419f-9c68-126fecb418fe\" (UID: \"20bdaf4e-ad15-419f-9c68-126fecb418fe\") "
Nov 03 14:38:33.406386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406260    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.406386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406300    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.406561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406544    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.406967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406944    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.407063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406973    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.407063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.406990    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.407063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.407005    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.407063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.407017    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20bdaf4e-ad15-419f-9c68-126fecb418fe-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.407937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.407915    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "kube-api-access-9tnn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:33.408019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.407942    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.408019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.407954    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:33.408163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.408145    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token" pod "20bdaf4e-ad15-419f-9c68-126fecb418fe" (UID: "20bdaf4e-ad15-419f-9c68-126fecb418fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:33.508195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.508155    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.508862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.508839    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.509108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.508774    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.509207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.509120    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.509363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.509344    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.509510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.509496    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.510004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.509701    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.510655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.509958    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.510865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.510847    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.511127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511111    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.511260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511116    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:33.511418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511400    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.511542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511530    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.511703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511676    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.511868 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511851    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.512004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511989    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.512137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512123    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.512280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512264    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.512423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512409    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.512545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512531    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.512665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512651    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685\") pod \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\" (UID: \"0c804f17-3e91-41ff-99f5-5a2ad073f3ed\") "
Nov 03 14:38:33.512797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512783    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket\") pod \"90630806-857f-41ea-8571-f0c8439c1fbe\" (UID: \"90630806-857f-41ea-8571-f0c8439c1fbe\") "
Nov 03 14:38:33.513565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.511870    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.513565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512185    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.513565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512668    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.513779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.512943    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.513779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.513537    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:33.514218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514199    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.514395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514378    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.514503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514490    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.514589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514577    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.514675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514664    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.514764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514753    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90630806-857f-41ea-8571-f0c8439c1fbe-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.514905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.514871    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tnn2\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-kube-api-access-9tnn2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.515019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.515004    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.515115 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.515105    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20bdaf4e-ad15-419f-9c68-126fecb418fe-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.515683 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.515650    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:33.515770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.515692    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:33.515986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.515955    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "kube-api-access-c5685". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:33.516102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.516034    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "kube-api-access-jj46r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:33.516524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.516506    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy" pod "0c804f17-3e91-41ff-99f5-5a2ad073f3ed" (UID: "0c804f17-3e91-41ff-99f5-5a2ad073f3ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.516631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.516601    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:33.517017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.517001    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:33.518165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.518146    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo" pod "90630806-857f-41ea-8571-f0c8439c1fbe" (UID: "90630806-857f-41ea-8571-f0c8439c1fbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:33.561024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.560985    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"]
Nov 03 14:38:33.561626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561605    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7f62a75a-1223-4f40-bccc-061e8000e822" containerName="payload-processing"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561627    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f62a75a-1223-4f40-bccc-061e8000e822" containerName="payload-processing"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561655    6481 cpu_manager.go:414] "containerMap: removing container" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561664    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561679    6481 cpu_manager.go:414] "containerMap: removing container" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561686    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561720    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy"
Nov 03 14:38:33.561750 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561727    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561756    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f1ec9411-5fb7-4467-afc8-e646daa8305c" containerName="revoke-keys"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561763    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ec9411-5fb7-4467-afc8-e646daa8305c" containerName="revoke-keys"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561895    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerName="istio-proxy"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561914    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ec9411-5fb7-4467-afc8-e646daa8305c" containerName="revoke-keys"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561924    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f62a75a-1223-4f40-bccc-061e8000e822" containerName="payload-processing"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561937    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerName="istio-proxy"
Nov 03 14:38:33.562128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.561948    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" containerName="istio-proxy"
Nov 03 14:38:33.567629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.567608    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.583593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.583570    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-331c9e-openshift-default-dockercfg-skzpv\""
Nov 03 14:38:33.584778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.584757    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"]
Nov 03 14:38:33.616043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616021    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616043    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616053    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616061    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616069    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616079    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616087    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616094    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616102    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5685\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-kube-api-access-c5685\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616110    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90630806-857f-41ea-8571-f0c8439c1fbe-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616118    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616128    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj46r\" (UniqueName: \"kubernetes.io/projected/90630806-857f-41ea-8571-f0c8439c1fbe-kube-api-access-jj46r\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616137    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90630806-857f-41ea-8571-f0c8439c1fbe-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.616215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.616145    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c804f17-3e91-41ff-99f5-5a2ad073f3ed-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720286    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720359    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m48s9\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720420    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720553    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720615    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720715    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720787    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720839    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.729254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.720887    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822127    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822178    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822213    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m48s9\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822286    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822377    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822418    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822491    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822548    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822577    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822592    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822766    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.822918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.822890    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.823287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.823265    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.823459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.823442    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.824546 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.824497    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.824797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.824780    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.842743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.842719    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.842911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.842892    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m48s9\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:33.881369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:33.881348    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:34.083420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.083387    6481 generic.go:356] "Generic (PLEG): container finished" podID="20bdaf4e-ad15-419f-9c68-126fecb418fe" containerID="a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f" exitCode=0
Nov 03 14:38:34.083560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.083461    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"
Nov 03 14:38:34.083560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.083494    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" event={"ID":"20bdaf4e-ad15-419f-9c68-126fecb418fe","Type":"ContainerDied","Data":"a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f"}
Nov 03 14:38:34.083560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.083557    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s" event={"ID":"20bdaf4e-ad15-419f-9c68-126fecb418fe","Type":"ContainerDied","Data":"53192d52c772d3a10bfcd34a193e78cebd98d3fe6bd77713837cf55a9b075133"}
Nov 03 14:38:34.083719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.083586    6481 scope.go:122] "RemoveContainer" containerID="a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f"
Nov 03 14:38:34.085352 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.085332    6481 generic.go:356] "Generic (PLEG): container finished" podID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" containerID="374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a" exitCode=0
Nov 03 14:38:34.085484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.085399    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"
Nov 03 14:38:34.085484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.085423    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" event={"ID":"0c804f17-3e91-41ff-99f5-5a2ad073f3ed","Type":"ContainerDied","Data":"374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a"}
Nov 03 14:38:34.085484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.085481    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls" event={"ID":"0c804f17-3e91-41ff-99f5-5a2ad073f3ed","Type":"ContainerDied","Data":"63d5a7581555052030dfa106396326df5f9083b04c9db806738d76694e1ea065"}
Nov 03 14:38:34.087193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.087171    6481 generic.go:356] "Generic (PLEG): container finished" podID="90630806-857f-41ea-8571-f0c8439c1fbe" containerID="62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5" exitCode=0
Nov 03 14:38:34.087335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.087315    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"
Nov 03 14:38:34.087429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.087409    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" event={"ID":"90630806-857f-41ea-8571-f0c8439c1fbe","Type":"ContainerDied","Data":"62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5"}
Nov 03 14:38:34.087552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.087460    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9" event={"ID":"90630806-857f-41ea-8571-f0c8439c1fbe","Type":"ContainerDied","Data":"b2cd4717f1a27558347a62fd7ecc4f30bbcdc701a190763309e391f87bc90722"}
Nov 03 14:38:34.101028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.101009    6481 scope.go:122] "RemoveContainer" containerID="a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f"
Nov 03 14:38:34.101433 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:34.101413    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f\": container with ID starting with a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f not found: ID does not exist" containerID="a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f"
Nov 03 14:38:34.101533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.101440    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f"} err="failed to get container status \"a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f\": rpc error: code = NotFound desc = could not find container \"a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f\": container with ID starting with a69b8800778336c28865a448a2ff7fa60ce1eb9a697f37123e2b368d0791b07f not found: ID does not exist"
Nov 03 14:38:34.101533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.101456    6481 scope.go:122] "RemoveContainer" containerID="374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a"
Nov 03 14:38:34.110787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.110757    6481 scope.go:122] "RemoveContainer" containerID="374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a"
Nov 03 14:38:34.111045 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:34.111028    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a\": container with ID starting with 374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a not found: ID does not exist" containerID="374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a"
Nov 03 14:38:34.111093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.111052    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a"} err="failed to get container status \"374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a\": rpc error: code = NotFound desc = could not find container \"374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a\": container with ID starting with 374d8f4b05c3a588d273409674670d09ed8510a3f7de5d92c4a5b2d733f2681a not found: ID does not exist"
Nov 03 14:38:34.111093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.111067    6481 scope.go:122] "RemoveContainer" containerID="62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5"
Nov 03 14:38:34.119705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.119684    6481 scope.go:122] "RemoveContainer" containerID="62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5"
Nov 03 14:38:34.119948 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:34.119928    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5\": container with ID starting with 62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5 not found: ID does not exist" containerID="62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5"
Nov 03 14:38:34.120013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.119957    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5"} err="failed to get container status \"62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5\": rpc error: code = NotFound desc = could not find container \"62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5\": container with ID starting with 62a36d56c6c01a599c83a5f46002e090fd44c5313d6e8b84085a9c30756490a5 not found: ID does not exist"
Nov 03 14:38:34.158733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.158681    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"]
Nov 03 14:38:34.159029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.158984    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy" containerID="cri-o://f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266" gracePeriod=30
Nov 03 14:38:34.165248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.165206    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"]
Nov 03 14:38:34.165471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.165441    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy" containerID="cri-o://258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03" gracePeriod=30
Nov 03 14:38:34.184902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.184870    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"]
Nov 03 14:38:34.186994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.186964    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:34.191752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.191727    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-brgls"]
Nov 03 14:38:34.215418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.215294    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"]
Nov 03 14:38:34.216190 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:34.216167    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef3669f_7f28_45a7_9e72_7b705e2bc8d5.slice/crio-21e8b4bb72f27face994c67de400c553570f942569b21c2bfd7b3cbfb4112656 WatchSource:0}: Error finding container 21e8b4bb72f27face994c67de400c553570f942569b21c2bfd7b3cbfb4112656: Status 404 returned error can't find the container with id 21e8b4bb72f27face994c67de400c553570f942569b21c2bfd7b3cbfb4112656
Nov 03 14:38:34.218392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.218349    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:34.218508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.218436    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:34.218508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.218478    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:38:34.232119 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.232090    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"]
Nov 03 14:38:34.235854 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.235830    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-86w7s"]
Nov 03 14:38:34.268045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.268010    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"]
Nov 03 14:38:34.275980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:34.275957    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-86c2e2-openshift-default-6f7db89749-74fq9"]
Nov 03 14:38:35.096484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.096448    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" event={"ID":"aef3669f-7f28-45a7-9e72-7b705e2bc8d5","Type":"ContainerStarted","Data":"15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9"}
Nov 03 14:38:35.096927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.096509    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" event={"ID":"aef3669f-7f28-45a7-9e72-7b705e2bc8d5","Type":"ContainerStarted","Data":"21e8b4bb72f27face994c67de400c553570f942569b21c2bfd7b3cbfb4112656"}
Nov 03 14:38:35.143980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.143951    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"]
Nov 03 14:38:35.249358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.248379    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podStartSLOduration=2.248359474 podStartE2EDuration="2.248359474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:35.171972429 +0000 UTC m=+1593.222003614" watchObservedRunningTime="2026-11-03 14:38:35.248359474 +0000 UTC m=+1593.298390662"
Nov 03 14:38:35.326756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.326705    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:35.537899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.537857    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:35.752249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.751872    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:35.890339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.887588    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:35.890339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.889481    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 14:38:35.890339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:35.889534    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 14:38:36.067160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.067127    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c804f17-3e91-41ff-99f5-5a2ad073f3ed" path="/var/lib/kubelet/pods/0c804f17-3e91-41ff-99f5-5a2ad073f3ed/volumes"
Nov 03 14:38:36.067613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.067598    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bdaf4e-ad15-419f-9c68-126fecb418fe" path="/var/lib/kubelet/pods/20bdaf4e-ad15-419f-9c68-126fecb418fe/volumes"
Nov 03 14:38:36.068011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.067999    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90630806-857f-41ea-8571-f0c8439c1fbe" path="/var/lib/kubelet/pods/90630806-857f-41ea-8571-f0c8439c1fbe/volumes"
Nov 03 14:38:36.431262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.430884    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:36.724307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.721673    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:36.882565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.882536    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 14:38:36.882792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.882770    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 14:38:36.937709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:36.937672    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:37.356512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:37.356480    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:37.575846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:37.575800    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:37.881818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:37.881786    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 14:38:37.882018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:37.881837    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 14:38:38.600133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:38.600103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:38.826493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:38.826456    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:38.882355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:38.882285    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 14:38:38.882355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:38.882333    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 14:38:39.076619 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:39.076586    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-386f6d33da21: secret "agc-cred-386f6d33da21" not found
Nov 03 14:38:39.076619 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:39.076613    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/e2e-cfl-d4022b-provider-cred: secret "e2e-cfl-d4022b-provider-cred" not found
Nov 03 14:38:39.076619 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:39.076624    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv: [secret "agc-cred-386f6d33da21" not found, secret "e2e-cfl-d4022b-provider-cred" not found]
Nov 03 14:38:39.076892 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:39.076675    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials podName:e831edb9-8e4a-4070-91ab-4f1709b64ff7 nodeName:}" failed. No retries permitted until 2026-11-03 14:38:55.076661973 +0000 UTC m=+1613.126693139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7") : [secret "agc-cred-386f6d33da21" not found, secret "e2e-cfl-d4022b-provider-cred" not found]
Nov 03 14:38:39.487343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.487319    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:38:39.528909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.528851    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:39.553149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.553127    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:38:39.582138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582106    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582171    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582198    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582277    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582320    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582351    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582400    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582414    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.582505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582465    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582495    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582528    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582576    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582608    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582697    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.582805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582732    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.582805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582737    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582850    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582896    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582912    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket\") pod \"51173fe0-38c4-4a2e-948c-f595deedce2b\" (UID: \"51173fe0-38c4-4a2e-948c-f595deedce2b\") "
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.582972    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.583006    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data\") pod \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\" (UID: \"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3\") "
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.583096    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.583130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.583121    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.583625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.583186    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:39.583625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.583355    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.583974    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584002    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584018    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584032    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584045    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584059    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584071    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.584726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584438    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.584726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.584521    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:39.585118 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.585076    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "kube-api-access-m9z57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:39.585412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.585367    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:39.585527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.585410    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:39.585696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.585666    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.585831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.585759    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:39.586662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.586633    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:39.586764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.586698    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "kube-api-access-c87lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:39.586883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.586863    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy" pod "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" (UID: "1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.587013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.586996    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy" pod "51173fe0-38c4-4a2e-948c-f595deedce2b" (UID: "51173fe0-38c4-4a2e-948c-f595deedce2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:39.685686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685654    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.685686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685681    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51173fe0-38c4-4a2e-948c-f595deedce2b-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.685686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685691    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685699    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87lm\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-kube-api-access-c87lm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685710    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685718    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685729    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685739    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685747    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685758    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51173fe0-38c4-4a2e-948c-f595deedce2b-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.686129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.685766    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9z57\" (UniqueName: \"kubernetes.io/projected/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3-kube-api-access-m9z57\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:39.759558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.758603    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"]
Nov 03 14:38:39.759558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.758804    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" containerID="cri-o://51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03" gracePeriod=30
Nov 03 14:38:39.771045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.771019    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"]
Nov 03 14:38:39.771270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.771210    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy" containerID="cri-o://bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c" gracePeriod=30
Nov 03 14:38:39.778054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.778029    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"]
Nov 03 14:38:39.778282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.778256    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy" containerID="cri-o://868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38" gracePeriod=30
Nov 03 14:38:39.896832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.890896    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Nov 03 14:38:39.896832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.890964    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Nov 03 14:38:39.943251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.942193    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:39.963325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:39.963293    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:38:40.147361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.147290    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" podUID="e831edb9-8e4a-4070-91ab-4f1709b64ff7" containerName="payload-processing" containerID="cri-o://55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba" gracePeriod=25
Nov 03 14:38:40.171586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.171027    6481 generic.go:356] "Generic (PLEG): container finished" podID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerID="f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266" exitCode=0
Nov 03 14:38:40.171586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.171257    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" event={"ID":"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3","Type":"ContainerDied","Data":"f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266"}
Nov 03 14:38:40.171586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.171320    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6" event={"ID":"1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3","Type":"ContainerDied","Data":"39aead63c7adbf854ac5baa669ce5e406c340eb3a93a90aba5bc0f3b9779a2bd"}
Nov 03 14:38:40.171586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.171352    6481 scope.go:122] "RemoveContainer" containerID="f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266"
Nov 03 14:38:40.171586 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.171522    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"
Nov 03 14:38:40.180686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.180597    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" event={"ID":"51173fe0-38c4-4a2e-948c-f595deedce2b","Type":"ContainerDied","Data":"258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03"}
Nov 03 14:38:40.180935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.180647    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"
Nov 03 14:38:40.181035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.180494    6481 generic.go:356] "Generic (PLEG): container finished" podID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerID="258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03" exitCode=0
Nov 03 14:38:40.181114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.181084    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg" event={"ID":"51173fe0-38c4-4a2e-948c-f595deedce2b","Type":"ContainerDied","Data":"9159bd7ce9885385a2f62a17cf3883ebb7ae353458a740830484ab17dc10359e"}
Nov 03 14:38:40.264945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.264894    6481 scope.go:122] "RemoveContainer" containerID="f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266"
Nov 03 14:38:40.265520 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:40.265357    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266\": container with ID starting with f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266 not found: ID does not exist" containerID="f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266"
Nov 03 14:38:40.265520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.265397    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266"} err="failed to get container status \"f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266\": rpc error: code = NotFound desc = could not find container \"f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266\": container with ID starting with f049c407f6bbd9e8c6fd675f8b980e0330ed9b09f6aacf4c610fc2b4132c0266 not found: ID does not exist"
Nov 03 14:38:40.265520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.265422    6481 scope.go:122] "RemoveContainer" containerID="258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03"
Nov 03 14:38:40.276813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.276614    6481 scope.go:122] "RemoveContainer" containerID="258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03"
Nov 03 14:38:40.276971 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:40.276946    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03\": container with ID starting with 258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03 not found: ID does not exist" containerID="258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03"
Nov 03 14:38:40.277051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.276977    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03"} err="failed to get container status \"258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03\": rpc error: code = NotFound desc = could not find container \"258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03\": container with ID starting with 258fb85ec26b9329b8ca542bed19e842a868d9c9250dfcce1e3abb557fa28e03 not found: ID does not exist"
Nov 03 14:38:40.285508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.285481    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"]
Nov 03 14:38:40.296337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.296298    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-5nzrg"]
Nov 03 14:38:40.345117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.345003    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"]
Nov 03 14:38:40.349922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.349893    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-2zrh6"]
Nov 03 14:38:40.363252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.359816    6481 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 14:38:40.363252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.359898    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 14:38:40.465797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.465775    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:40.562815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.562787    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials\") pod \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") "
Nov 03 14:38:40.562985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.562843    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g\") pod \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") "
Nov 03 14:38:40.562985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.562915    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume\") pod \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") "
Nov 03 14:38:40.562985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.562983    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume\") pod \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\" (UID: \"e831edb9-8e4a-4070-91ab-4f1709b64ff7\") "
Nov 03 14:38:40.563366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.563343    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume" pod "e831edb9-8e4a-4070-91ab-4f1709b64ff7" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:40.563480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.563458    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume" pod "e831edb9-8e4a-4070-91ab-4f1709b64ff7" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:40.565507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.565482    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials" pod "e831edb9-8e4a-4070-91ab-4f1709b64ff7" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:40.565589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.565517    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g" pod "e831edb9-8e4a-4070-91ab-4f1709b64ff7" (UID: "e831edb9-8e4a-4070-91ab-4f1709b64ff7"). InnerVolumeSpecName "kube-api-access-jkz6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:40.664370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.664334    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:40.664370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.664360    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:40.664370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.664371    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkz6g\" (UniqueName: \"kubernetes.io/projected/e831edb9-8e4a-4070-91ab-4f1709b64ff7-kube-api-access-jkz6g\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:40.664696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.664379    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e831edb9-8e4a-4070-91ab-4f1709b64ff7-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:40.798311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:40.798268    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:41.054885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.054806    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:41.055324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.055300    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:41.056193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.056171    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:38:41.192601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.192564    6481 generic.go:356] "Generic (PLEG): container finished" podID="e831edb9-8e4a-4070-91ab-4f1709b64ff7" containerID="55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba" exitCode=0
Nov 03 14:38:41.192781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.192631    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"
Nov 03 14:38:41.192781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.192649    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" event={"ID":"e831edb9-8e4a-4070-91ab-4f1709b64ff7","Type":"ContainerDied","Data":"55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba"}
Nov 03 14:38:41.192781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.192700    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv" event={"ID":"e831edb9-8e4a-4070-91ab-4f1709b64ff7","Type":"ContainerDied","Data":"9088039df70e62cc9bf9fa1f055cc4e6e698571d773de619c07fa486cafd5483"}
Nov 03 14:38:41.192781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.192726    6481 scope.go:122] "RemoveContainer" containerID="55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba"
Nov 03 14:38:41.204437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.204361    6481 scope.go:122] "RemoveContainer" containerID="55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba"
Nov 03 14:38:41.204708 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:41.204670    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba\": container with ID starting with 55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba not found: ID does not exist" containerID="55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba"
Nov 03 14:38:41.204862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.204833    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba"} err="failed to get container status \"55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba\": rpc error: code = NotFound desc = could not find container \"55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba\": container with ID starting with 55cad457a036d0918acc51902f6329c0ff719a59f1ba7591e0e3b6ae71b85dba not found: ID does not exist"
Nov 03 14:38:41.249397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.249363    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"]
Nov 03 14:38:41.260560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:41.260516    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-7fbfmnmlv"]
Nov 03 14:38:42.067026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:42.066996    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" path="/var/lib/kubelet/pods/1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3/volumes"
Nov 03 14:38:42.067510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:42.067494    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" path="/var/lib/kubelet/pods/51173fe0-38c4-4a2e-948c-f595deedce2b/volumes"
Nov 03 14:38:42.067883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:42.067871    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e831edb9-8e4a-4070-91ab-4f1709b64ff7" path="/var/lib/kubelet/pods/e831edb9-8e4a-4070-91ab-4f1709b64ff7/volumes"
Nov 03 14:38:42.270933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:42.270899    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:38:42.271086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:42.270967    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:38:42.790775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:42.790745    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:44.040710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:44.040659    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"]
Nov 03 14:38:44.052032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:44.052008    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13e953b1-795jf"]
Nov 03 14:38:44.067349 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:44.067323    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1852b2b0-30e9-4d6e-981b-bb6371406529" path="/var/lib/kubelet/pods/1852b2b0-30e9-4d6e-981b-bb6371406529/volumes"
Nov 03 14:38:44.626323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:44.626288    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:44.846869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:44.846843    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:45.053191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.053168    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:38:45.079211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.079190    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:38:45.086716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.086698    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:38:45.115134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115108    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115154    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115276    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115312    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115362    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115384    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115478    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115523    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.115732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115709    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.115732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115724    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:45.115846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115787    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.115846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.115840    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket\") pod \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\" (UID: \"6da9b341-ead3-421e-95cf-ce7f1e1730c6\") "
Nov 03 14:38:45.116137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116113    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.116255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116155    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.116806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116788    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.116806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116806    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.116973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116817    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.116973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116831    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.116973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.116838    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.117400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.117380    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.117535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.117522    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:45.117822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.117802    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "kube-api-access-b7sgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:45.117822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.117811    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo" pod "6da9b341-ead3-421e-95cf-ce7f1e1730c6" (UID: "6da9b341-ead3-421e-95cf-ce7f1e1730c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:45.223472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223440    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.223656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223504    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.223656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223532    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.223656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223578    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.223836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223661    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.223836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223697    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.223836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223743    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.223836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223782    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.223836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223805    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223855    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223885    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223913    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223932    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223950    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.223975    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224015    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224050    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs\") pod \"5255b506-ba1d-486d-a398-8ea873423e32\" (UID: \"5255b506-ba1d-486d-a398-8ea873423e32\") "
Nov 03 14:38:45.224113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224078    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token\") pod \"35a634ab-4e7b-4cac-8d9b-71663047f985\" (UID: \"35a634ab-4e7b-4cac-8d9b-71663047f985\") "
Nov 03 14:38:45.224574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224489    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.224574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224504    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.224574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224515    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7sgz\" (UniqueName: \"kubernetes.io/projected/6da9b341-ead3-421e-95cf-ce7f1e1730c6-kube-api-access-b7sgz\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.224574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224524    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6da9b341-ead3-421e-95cf-ce7f1e1730c6-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.224763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.224684    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.225101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.225073    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.225101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.225088    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.225356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.225090    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.226205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.226140    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.226708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.226682    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.227516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.227483    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:45.227804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.227776    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:45.228042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.228018    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.228136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.228117    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:45.228441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.228338    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.229299 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.229272    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.230114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.230094    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "kube-api-access-v65cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:45.230286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.230262    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "kube-api-access-pnk2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:45.230604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.230579    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo" pod "5255b506-ba1d-486d-a398-8ea873423e32" (UID: "5255b506-ba1d-486d-a398-8ea873423e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:45.230737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.230599    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:38:45.230737 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.230662    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:45.232069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.232044    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy" pod "35a634ab-4e7b-4cac-8d9b-71663047f985" (UID: "35a634ab-4e7b-4cac-8d9b-71663047f985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:38:45.232173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.232158    6481 generic.go:356] "Generic (PLEG): container finished" podID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerID="bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c" exitCode=0
Nov 03 14:38:45.232396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.232381    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"
Nov 03 14:38:45.233011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.232991    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" event={"ID":"35a634ab-4e7b-4cac-8d9b-71663047f985","Type":"ContainerDied","Data":"bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c"}
Nov 03 14:38:45.233116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.233038    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk" event={"ID":"35a634ab-4e7b-4cac-8d9b-71663047f985","Type":"ContainerDied","Data":"7f01827262a5e9caf7f3af9a941680fca0fe3520359a2fa4a3b96565a763261f"}
Nov 03 14:38:45.233116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.233061    6481 scope.go:122] "RemoveContainer" containerID="bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c"
Nov 03 14:38:45.241274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.237530    6481 generic.go:356] "Generic (PLEG): container finished" podID="5255b506-ba1d-486d-a398-8ea873423e32" containerID="868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38" exitCode=0
Nov 03 14:38:45.241274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.237723    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"
Nov 03 14:38:45.241274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.238310    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" event={"ID":"5255b506-ba1d-486d-a398-8ea873423e32","Type":"ContainerDied","Data":"868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38"}
Nov 03 14:38:45.241274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.238361    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs" event={"ID":"5255b506-ba1d-486d-a398-8ea873423e32","Type":"ContainerDied","Data":"0fda3c4652c7b082a01f7346da592f3ac0fe0cb917640f327dd7d994ddc99811"}
Nov 03 14:38:45.252601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.251305    6481 generic.go:356] "Generic (PLEG): container finished" podID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerID="51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03" exitCode=0
Nov 03 14:38:45.252601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.251372    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" event={"ID":"6da9b341-ead3-421e-95cf-ce7f1e1730c6","Type":"ContainerDied","Data":"51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03"}
Nov 03 14:38:45.252601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.251399    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"
Nov 03 14:38:45.252601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.251410    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm" event={"ID":"6da9b341-ead3-421e-95cf-ce7f1e1730c6","Type":"ContainerDied","Data":"bf61f4e94415633d8f920f57ab53dc5a743e8af727270a7c1f7d5795ed6946c6"}
Nov 03 14:38:45.258207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.258186    6481 scope.go:122] "RemoveContainer" containerID="bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c"
Nov 03 14:38:45.259172 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:45.259145    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c\": container with ID starting with bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c not found: ID does not exist" containerID="bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c"
Nov 03 14:38:45.259311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.259178    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c"} err="failed to get container status \"bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c\": rpc error: code = NotFound desc = could not find container \"bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c\": container with ID starting with bfed1c62d8f2e9036a49560048c7a4d35e9086467d60f38f9ccfd9231d87093c not found: ID does not exist"
Nov 03 14:38:45.259311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.259199    6481 scope.go:122] "RemoveContainer" containerID="868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38"
Nov 03 14:38:45.272059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.272033    6481 scope.go:122] "RemoveContainer" containerID="868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38"
Nov 03 14:38:45.272360 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:45.272340    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38\": container with ID starting with 868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38 not found: ID does not exist" containerID="868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38"
Nov 03 14:38:45.272426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.272365    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38"} err="failed to get container status \"868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38\": rpc error: code = NotFound desc = could not find container \"868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38\": container with ID starting with 868042b0ba9544492c3d2e5a9bf2becb106f310844b91d668cf959f4aff2ad38 not found: ID does not exist"
Nov 03 14:38:45.272426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.272381    6481 scope.go:122] "RemoveContainer" containerID="51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03"
Nov 03 14:38:45.285574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.285557    6481 scope.go:122] "RemoveContainer" containerID="7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"
Nov 03 14:38:45.295460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.295368    6481 scope.go:122] "RemoveContainer" containerID="51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03"
Nov 03 14:38:45.295980 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:45.295849    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03\": container with ID starting with 51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03 not found: ID does not exist" containerID="51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03"
Nov 03 14:38:45.295980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.295880    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03"} err="failed to get container status \"51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03\": rpc error: code = NotFound desc = could not find container \"51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03\": container with ID starting with 51fd8097e95778607da424a1e3c465560558c81c2b3343c40f1c31f2078eee03 not found: ID does not exist"
Nov 03 14:38:45.295980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.295901    6481 scope.go:122] "RemoveContainer" containerID="7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"
Nov 03 14:38:45.296837 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:45.296761    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821\": container with ID starting with 7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821 not found: ID does not exist" containerID="7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"
Nov 03 14:38:45.296837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.296791    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821"} err="failed to get container status \"7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821\": rpc error: code = NotFound desc = could not find container \"7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821\": container with ID starting with 7df5786fc407ae36e2e8a6ac4576be37cb4a8b445d1758b1b6ad24ce9d064821 not found: ID does not exist"
Nov 03 14:38:45.306177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.306153    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"]
Nov 03 14:38:45.312145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.312112    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-vcvvm"]
Nov 03 14:38:45.325137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325108    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325135    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325147    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5255b506-ba1d-486d-a398-8ea873423e32-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325160    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v65cd\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-kube-api-access-v65cd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325173    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325189    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnk2z\" (UniqueName: \"kubernetes.io/projected/5255b506-ba1d-486d-a398-8ea873423e32-kube-api-access-pnk2z\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325201    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325215    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325243    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325257    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325285    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325298    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5255b506-ba1d-486d-a398-8ea873423e32-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325310    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325324    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a634ab-4e7b-4cac-8d9b-71663047f985-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325339    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a634ab-4e7b-4cac-8d9b-71663047f985-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325354    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325366    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5255b506-ba1d-486d-a398-8ea873423e32-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.325752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.325379    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a634ab-4e7b-4cac-8d9b-71663047f985-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:45.347528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.347502    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"]
Nov 03 14:38:45.352990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.352968    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-988gs"]
Nov 03 14:38:45.388390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.388370    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"]
Nov 03 14:38:45.390940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:45.390920    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2d0da2-openshift-default-7cc57b7979-68ptk"]
Nov 03 14:38:46.068535 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:46.068463    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" path="/var/lib/kubelet/pods/35a634ab-4e7b-4cac-8d9b-71663047f985/volumes"
Nov 03 14:38:46.068912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:46.068898    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5255b506-ba1d-486d-a398-8ea873423e32" path="/var/lib/kubelet/pods/5255b506-ba1d-486d-a398-8ea873423e32/volumes"
Nov 03 14:38:46.069327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:46.069314    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" path="/var/lib/kubelet/pods/6da9b341-ead3-421e-95cf-ce7f1e1730c6/volumes"
Nov 03 14:38:46.906167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:46.906134    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:48.896211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:48.894725    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:48.966449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:48.966425    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:38:48.987806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:48.987778    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:38:49.064742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.064712    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume\") pod \"1bc877e4-adb7-4022-92bf-3dacf7e50950\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") "
Nov 03 14:38:49.064924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.064791    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume\") pod \"3a92b45a-231f-4e27-a401-bed7dbf93732\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") "
Nov 03 14:38:49.064924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.064841    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w\" (UniqueName: \"kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w\") pod \"1bc877e4-adb7-4022-92bf-3dacf7e50950\" (UID: \"1bc877e4-adb7-4022-92bf-3dacf7e50950\") "
Nov 03 14:38:49.064924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.064899    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn\" (UniqueName: \"kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn\") pod \"3a92b45a-231f-4e27-a401-bed7dbf93732\" (UID: \"3a92b45a-231f-4e27-a401-bed7dbf93732\") "
Nov 03 14:38:49.065245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.065204    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume" pod "3a92b45a-231f-4e27-a401-bed7dbf93732" (UID: "3a92b45a-231f-4e27-a401-bed7dbf93732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:49.065376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.065359    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a92b45a-231f-4e27-a401-bed7dbf93732-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:49.066019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.065992    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume" pod "1bc877e4-adb7-4022-92bf-3dacf7e50950" (UID: "1bc877e4-adb7-4022-92bf-3dacf7e50950"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:38:49.067786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.067759    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w" pod "1bc877e4-adb7-4022-92bf-3dacf7e50950" (UID: "1bc877e4-adb7-4022-92bf-3dacf7e50950"). InnerVolumeSpecName "kube-api-access-5977w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:49.068577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.068540    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn" pod "3a92b45a-231f-4e27-a401-bed7dbf93732" (UID: "3a92b45a-231f-4e27-a401-bed7dbf93732"). InnerVolumeSpecName "kube-api-access-n4lzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:38:49.167050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.167009    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bc877e4-adb7-4022-92bf-3dacf7e50950-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:49.167050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.167045    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5977w\" (UniqueName: \"kubernetes.io/projected/1bc877e4-adb7-4022-92bf-3dacf7e50950-kube-api-access-5977w\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:49.167050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.167059    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4lzn\" (UniqueName: \"kubernetes.io/projected/3a92b45a-231f-4e27-a401-bed7dbf93732-kube-api-access-n4lzn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:38:49.288942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.288849    6481 generic.go:356] "Generic (PLEG): container finished" podID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerID="5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407" exitCode=0
Nov 03 14:38:49.288942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.288896    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"
Nov 03 14:38:49.289178 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.288937    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" event={"ID":"1bc877e4-adb7-4022-92bf-3dacf7e50950","Type":"ContainerDied","Data":"5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407"}
Nov 03 14:38:49.289178 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.288996    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45" event={"ID":"1bc877e4-adb7-4022-92bf-3dacf7e50950","Type":"ContainerDied","Data":"d7dc399d36e0605c6fc8589564da06b65f55f62541c499ac7df29d4c4381bcc7"}
Nov 03 14:38:49.289178 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.289026    6481 scope.go:122] "RemoveContainer" containerID="5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407"
Nov 03 14:38:49.290502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.290476    6481 generic.go:356] "Generic (PLEG): container finished" podID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerID="e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f" exitCode=0
Nov 03 14:38:49.290637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.290518    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"
Nov 03 14:38:49.290637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.290560    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" event={"ID":"3a92b45a-231f-4e27-a401-bed7dbf93732","Type":"ContainerDied","Data":"e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f"}
Nov 03 14:38:49.290637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.290602    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds" event={"ID":"3a92b45a-231f-4e27-a401-bed7dbf93732","Type":"ContainerDied","Data":"49e5b9a6aef41f2224beb9cd8365953364c1eaa6be070a34d15a61f7bd727bf1"}
Nov 03 14:38:49.299826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.299767    6481 scope.go:122] "RemoveContainer" containerID="5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407"
Nov 03 14:38:49.300088 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:49.300063    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407\": container with ID starting with 5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407 not found: ID does not exist" containerID="5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407"
Nov 03 14:38:49.300167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.300098    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407"} err="failed to get container status \"5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407\": rpc error: code = NotFound desc = could not find container \"5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407\": container with ID starting with 5c5c887cf0dcc9b4eaba03a194f48921001de9c26cf196da27dbc44fe9eb2407 not found: ID does not exist"
Nov 03 14:38:49.300167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.300115    6481 scope.go:122] "RemoveContainer" containerID="e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f"
Nov 03 14:38:49.308912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.308899    6481 scope.go:122] "RemoveContainer" containerID="e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f"
Nov 03 14:38:49.309164 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:49.309145    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f\": container with ID starting with e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f not found: ID does not exist" containerID="e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f"
Nov 03 14:38:49.309201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.309172    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f"} err="failed to get container status \"e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f\": rpc error: code = NotFound desc = could not find container \"e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f\": container with ID starting with e640ae9c14da2c672fe48a8f52fbcc4bcba74a7e77c6ff516136712e4675bf9f not found: ID does not exist"
Nov 03 14:38:49.347806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.347784    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"]
Nov 03 14:38:49.359247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.359213    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2d0da2-7d857d58ff-xrj45"]
Nov 03 14:38:49.392530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.392499    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"]
Nov 03 14:38:49.395104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:49.395083    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2d0da2-6d9558984f-bk4ds"]
Nov 03 14:38:50.066769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:50.066694    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" path="/var/lib/kubelet/pods/1bc877e4-adb7-4022-92bf-3dacf7e50950/volumes"
Nov 03 14:38:50.067133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:50.067060    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" path="/var/lib/kubelet/pods/3a92b45a-231f-4e27-a401-bed7dbf93732/volumes"
Nov 03 14:38:50.985133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:50.985072    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:52.936361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:52.936329    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:53.162395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:53.162362    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:53.982431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:53.982402    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:54.212413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:54.212381    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:55.055337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.055309    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:55.572140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.572058    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:55.722029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.721997    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"]
Nov 03 14:38:55.722504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722489    6481 cpu_manager.go:414] "containerMap: removing container" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy"
Nov 03 14:38:55.722554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722505    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy"
Nov 03 14:38:55.722554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722519    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy"
Nov 03 14:38:55.722554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722524    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy"
Nov 03 14:38:55.722554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722540    6481 cpu_manager.go:414] "containerMap: removing container" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy"
Nov 03 14:38:55.722554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722545    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722559    6481 cpu_manager.go:414] "containerMap: removing container" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerName="payload-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722564    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerName="payload-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722571    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722575    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722582    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerName="payload-pre-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722587    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerName="payload-pre-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722596    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722601    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722607    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722612    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722620    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e831edb9-8e4a-4070-91ab-4f1709b64ff7" containerName="payload-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722625    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e831edb9-8e4a-4070-91ab-4f1709b64ff7" containerName="payload-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722696    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="51173fe0-38c4-4a2e-948c-f595deedce2b" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722706    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a92b45a-231f-4e27-a401-bed7dbf93732" containerName="payload-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722713    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722720    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc877e4-adb7-4022-92bf-3dacf7e50950" containerName="payload-pre-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722727    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5255b506-ba1d-486d-a398-8ea873423e32" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722731    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da9b341-ead3-421e-95cf-ce7f1e1730c6" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722737    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e831edb9-8e4a-4070-91ab-4f1709b64ff7" containerName="payload-processing"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722745    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a634ab-4e7b-4cac-8d9b-71663047f985" containerName="istio-proxy"
Nov 03 14:38:55.722757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.722751    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0b0ce3-9e5f-4670-9af0-53a27c4b9ec3" containerName="istio-proxy"
Nov 03 14:38:55.726153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.726136    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.745501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.745474    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"]
Nov 03 14:38:55.754402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.754364    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-88b8295771ff\""
Nov 03 14:38:55.754555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.754475    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 14:38:55.754688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.754568    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pz6cz\""
Nov 03 14:38:55.754688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.754617    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 14:38:55.756784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.756764    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 14:38:55.756922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.756896    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 14:38:55.844912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.844418    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.844912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.844521    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsqz\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.844912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.844589    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.844912 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.844700    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.959355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.959312    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.959552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.959362    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsqz\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.959552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.959465    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.959552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.959533    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.960185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.960158    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.960363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.960185    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.961956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.961930    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:55.980005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:55.979978    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsqz\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz\") pod \"payload-processing-external-model-bf9485bb6-jrz9d\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") " pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:56.039621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:56.039587    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:56.344196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:56.344161    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"]
Nov 03 14:38:56.347497 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:38:56.347436    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c562dae_c343_46b6_b08b_03bf89313f4b.slice/crio-118a36069de9bf1c6dff27e1c8dcb482a48a0f99a1a938ed3d9238a25c5a4b5d WatchSource:0}: Error finding container 118a36069de9bf1c6dff27e1c8dcb482a48a0f99a1a938ed3d9238a25c5a4b5d: Status 404 returned error can't find the container with id 118a36069de9bf1c6dff27e1c8dcb482a48a0f99a1a938ed3d9238a25c5a4b5d
Nov 03 14:38:57.021262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:57.021215    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:57.351088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:57.350998    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d" event={"ID":"7c562dae-c343-46b6-b08b-03bf89313f4b","Type":"ContainerStarted","Data":"87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d"}
Nov 03 14:38:57.351545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:57.351105    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d" event={"ID":"7c562dae-c343-46b6-b08b-03bf89313f4b","Type":"ContainerStarted","Data":"118a36069de9bf1c6dff27e1c8dcb482a48a0f99a1a938ed3d9238a25c5a4b5d"}
Nov 03 14:38:57.351545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:57.351261    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:38:57.396246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:57.396176    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d" podStartSLOduration=2.396158865 podStartE2EDuration="2.396158865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:57.395459167 +0000 UTC m=+1615.445490351" watchObservedRunningTime="2026-11-03 14:38:57.396158865 +0000 UTC m=+1615.446190051"
Nov 03 14:38:57.405681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:57.405647    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:38:58.024414 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:38:58.024368    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:38:58.044775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:58.044744    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"]
Nov 03 14:38:58.055957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:58.055925    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-577947f1-x5dbm"]
Nov 03 14:38:58.071010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:58.070974    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="080bf727-2419-432e-a67c-c0980df7f81a" path="/var/lib/kubelet/pods/080bf727-2419-432e-a67c-c0980df7f81a/volumes"
Nov 03 14:38:59.240436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:59.240405    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:38:59.471350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:38:59.471039    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:01.440253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:01.440203    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:03.506405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:03.506372    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:05.508725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:05.508694    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:06.999479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:06.999442    6481 scope.go:122] "RemoveContainer" containerID="d0ce1a5a6e54c419c7399c1641c6e32f4dfd16d4aa106d4ee2a4d649126ac8a9"
Nov 03 14:39:07.012303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.012279    6481 scope.go:122] "RemoveContainer" containerID="3ae2a0b6e7e8136aba9799a1e1dba958471b2ac1eb12ee9e8bea8ce76884e091"
Nov 03 14:39:07.020843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.020819    6481 scope.go:122] "RemoveContainer" containerID="e20772591d9a6c553e868a302065793bf03c562ace1b8b24882db33fecca8288"
Nov 03 14:39:07.029699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.029671    6481 scope.go:122] "RemoveContainer" containerID="ed70e71e2569dad4bd94dc559bd2a14b4a2d4bf0523c74b238ebc93e8531f2e3"
Nov 03 14:39:07.038086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.038066    6481 scope.go:122] "RemoveContainer" containerID="930ef76828ca7304bd89b8020e2cf6e0adbcb694bce50e5c58306ebddfffda6c"
Nov 03 14:39:07.046441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.046415    6481 scope.go:122] "RemoveContainer" containerID="9718c398e11d326c7b499b59069ffb2f12725e29e0caaa2fd1a77d8b50444085"
Nov 03 14:39:07.054290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.054270    6481 scope.go:122] "RemoveContainer" containerID="c6f8b81808c1108697b0929017028c9c808198e69729d4778ffd9fac708fc9ce"
Nov 03 14:39:07.061946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.061923    6481 scope.go:122] "RemoveContainer" containerID="757faa96cbc2952c460623ce24bc1efbb03c856bdb6dc5a859199489803e01fc"
Nov 03 14:39:07.069731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.069714    6481 scope.go:122] "RemoveContainer" containerID="c2b7da830662cd22012ffbd3797eb25ec8cf5e9759b874f703eb8304f309e36a"
Nov 03 14:39:07.077550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.077524    6481 scope.go:122] "RemoveContainer" containerID="0e6fc7ba8386c436758661c3a9c232937b537debfd4da2416ae19fdd216e0fa1"
Nov 03 14:39:07.497669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:07.497618    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:08.362089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:08.362062    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:39:09.420165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:09.420108    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:11.401479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:11.401452    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:13.442712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:13.442637    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:14.678525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:14.678491    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:39:15.513157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:15.513121    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:17.796253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:17.795703    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:18.173249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:18.172376    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:39:18.173249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:18.172446    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:39:18.220252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:18.218366    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:39:18.220252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:18.218427    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:39:18.301256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:18.300356    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:39:18.301256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:18.300426    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:39:20.121928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:20.121895    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:21.963898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:21.963866    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"]
Nov 03 14:39:21.971340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:21.971320    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:21.989398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:21.989371    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"]
Nov 03 14:39:21.997329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:21.997304    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:39:21.997906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:21.997878    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:39:21.997906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:21.997894    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:39:22.001209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.001192    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:39:22.055353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.055315    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.055536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.055362    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.055536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.055507    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2dbc\" (UniqueName: \"kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.111302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.111267    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:22.156220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.156192    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2dbc\" (UniqueName: \"kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.156400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.156330    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.156400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.156348    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.156933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.156913    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.158599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.158582    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.177392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.177371    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2dbc\" (UniqueName: \"kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc\") pod \"maas-api-revoke-sub-703f6a22-gm9js\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.285354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.285294    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:22.586336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:22.586310    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"]
Nov 03 14:39:22.587268 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:39:22.587219    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e93ed3a_bd20_4a2c_9be6_d41ce1f46ffc.slice/crio-9a3a7d408718e6f516ef88b152a8674e30f7615da3c63c194304b1b03e248b16 WatchSource:0}: Error finding container 9a3a7d408718e6f516ef88b152a8674e30f7615da3c63c194304b1b03e248b16: Status 404 returned error can't find the container with id 9a3a7d408718e6f516ef88b152a8674e30f7615da3c63c194304b1b03e248b16
Nov 03 14:39:23.570367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:23.570338    6481 generic.go:356] "Generic (PLEG): container finished" podID="0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" containerID="94884579839f19dd673d9e2680f01bfc76d4c6f19915fed8e608c87fe091ff8a" exitCode=0
Nov 03 14:39:23.570714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:23.570428    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js" event={"ID":"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc","Type":"ContainerDied","Data":"94884579839f19dd673d9e2680f01bfc76d4c6f19915fed8e608c87fe091ff8a"}
Nov 03 14:39:23.570714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:23.570485    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js" event={"ID":"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc","Type":"ContainerStarted","Data":"9a3a7d408718e6f516ef88b152a8674e30f7615da3c63c194304b1b03e248b16"}
Nov 03 14:39:24.109772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.109741    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:24.332513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.332485    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:24.724135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.724110    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:24.883584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.883511    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc\" (UniqueName: \"kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc\") pod \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") "
Nov 03 14:39:24.883584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.883568    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca\") pod \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") "
Nov 03 14:39:24.883771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.883684    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca\") pod \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\" (UID: \"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc\") "
Nov 03 14:39:24.883936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.883911    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca" pod "0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" (UID: "0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:39:24.884060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.884006    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:24.885646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.885625    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca" pod "0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" (UID: "0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:39:24.885710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.885644    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc" pod "0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" (UID: "0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc"). InnerVolumeSpecName "kube-api-access-p2dbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:39:24.985499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.985470    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:24.985499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:24.985494    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2dbc\" (UniqueName: \"kubernetes.io/projected/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc-kube-api-access-p2dbc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:25.584485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:25.584458    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"
Nov 03 14:39:25.584682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:25.584497    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js" event={"ID":"0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc","Type":"ContainerDied","Data":"9a3a7d408718e6f516ef88b152a8674e30f7615da3c63c194304b1b03e248b16"}
Nov 03 14:39:25.584682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:25.584553    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a3a7d408718e6f516ef88b152a8674e30f7615da3c63c194304b1b03e248b16"
Nov 03 14:39:26.103013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:26.102973    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:27.985196 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:39:27.985148    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:39:28.098641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:28.098611    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:29.218874 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:29.218842    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"]
Nov 03 14:39:29.246852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:29.246822    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"]
Nov 03 14:39:29.434543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:29.434512    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:30.126049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:30.126019    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:30.346062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:30.346035    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:31.901030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:31.900991    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:39:32.106312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:32.106280    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:33.043079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:33.043050    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"]
Nov 03 14:39:33.054384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:33.054357    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b0262a5-fv2x4"]
Nov 03 14:39:34.066303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:34.066275    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7769d7-f0a3-49aa-aeef-ab8d49bebad9" path="/var/lib/kubelet/pods/1a7769d7-f0a3-49aa-aeef-ab8d49bebad9/volumes"
Nov 03 14:39:34.125611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:34.125582    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:35.464140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.464108    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"]
Nov 03 14:39:35.464610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.464597    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" containerName="revoke-keys"
Nov 03 14:39:35.464661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.464611    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" containerName="revoke-keys"
Nov 03 14:39:35.464704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.464693    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" containerName="revoke-keys"
Nov 03 14:39:35.469132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.469116    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.484818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.484801    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:39:35.495698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.495676    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:39:35.495842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.495822    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:39:35.495918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.495744    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:39:35.499932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.499906    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"]
Nov 03 14:39:35.599058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.599027    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.599253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.599154    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.599253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.599191    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6t7\" (UniqueName: \"kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.700130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.700096    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.700317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.700184    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.700317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.700209    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6t7\" (UniqueName: \"kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.700852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.700826    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.702480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.702458    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.721940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.721873    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6t7\" (UniqueName: \"kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7\") pod \"maas-api-revoke-sub-96fa1109-8qbwj\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:35.782715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:35.782688    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:36.087293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.087263    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"]
Nov 03 14:39:36.088798 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:39:36.088771    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9621b6_465d_4b17_8f92_15d830277191.slice/crio-70ee59143b192d1612ef3b4404d19a020cfa9c7d445eaaa83f5c603c2bffdda0 WatchSource:0}: Error finding container 70ee59143b192d1612ef3b4404d19a020cfa9c7d445eaaa83f5c603c2bffdda0: Status 404 returned error can't find the container with id 70ee59143b192d1612ef3b4404d19a020cfa9c7d445eaaa83f5c603c2bffdda0
Nov 03 14:39:36.200332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.200304    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"]
Nov 03 14:39:36.204473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.204436    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:36.446118 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.446091    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:36.663929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.663847    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj" event={"ID":"1f9621b6-465d-4b17-8f92-15d830277191","Type":"ContainerStarted","Data":"90a099c9f17a2103d9d9af4037edbd7117a4ac1ae445820dc85df72e626b99a9"}
Nov 03 14:39:36.663929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.663926    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj" event={"ID":"1f9621b6-465d-4b17-8f92-15d830277191","Type":"ContainerStarted","Data":"70ee59143b192d1612ef3b4404d19a020cfa9c7d445eaaa83f5c603c2bffdda0"}
Nov 03 14:39:36.704423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:36.704345    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj" podStartSLOduration=1.704325124 podStartE2EDuration="1.704325124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:39:36.702952131 +0000 UTC m=+1654.752983314" watchObservedRunningTime="2026-11-03 14:39:36.704325124 +0000 UTC m=+1654.754356308"
Nov 03 14:39:37.674157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:37.674113    6481 generic.go:356] "Generic (PLEG): container finished" podID="1f9621b6-465d-4b17-8f92-15d830277191" containerID="90a099c9f17a2103d9d9af4037edbd7117a4ac1ae445820dc85df72e626b99a9" exitCode=0
Nov 03 14:39:37.674604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:37.674185    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj" event={"ID":"1f9621b6-465d-4b17-8f92-15d830277191","Type":"ContainerDied","Data":"90a099c9f17a2103d9d9af4037edbd7117a4ac1ae445820dc85df72e626b99a9"}
Nov 03 14:39:38.389192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:38.389139    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:38.618060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:38.618029    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:38.875844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:38.875811    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:38.950891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:38.950781    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:38.954876 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:38.954852    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca\") pod \"1f9621b6-465d-4b17-8f92-15d830277191\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") "
Nov 03 14:39:38.955321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:38.955296    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca" pod "1f9621b6-465d-4b17-8f92-15d830277191" (UID: "1f9621b6-465d-4b17-8f92-15d830277191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:39:39.055739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.055709    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca\") pod \"1f9621b6-465d-4b17-8f92-15d830277191\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") "
Nov 03 14:39:39.055913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.055759    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7\" (UniqueName: \"kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7\") pod \"1f9621b6-465d-4b17-8f92-15d830277191\" (UID: \"1f9621b6-465d-4b17-8f92-15d830277191\") "
Nov 03 14:39:39.056113 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.056099    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:39.057799 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.057771    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca" pod "1f9621b6-465d-4b17-8f92-15d830277191" (UID: "1f9621b6-465d-4b17-8f92-15d830277191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:39:39.057941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.057909    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7" pod "1f9621b6-465d-4b17-8f92-15d830277191" (UID: "1f9621b6-465d-4b17-8f92-15d830277191"). InnerVolumeSpecName "kube-api-access-7c6t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:39:39.156988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.156954    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f9621b6-465d-4b17-8f92-15d830277191-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:39.156988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.156987    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6t7\" (UniqueName: \"kubernetes.io/projected/1f9621b6-465d-4b17-8f92-15d830277191-kube-api-access-7c6t7\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:39.412549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.412500    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:39.709696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.709602    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj" event={"ID":"1f9621b6-465d-4b17-8f92-15d830277191","Type":"ContainerDied","Data":"70ee59143b192d1612ef3b4404d19a020cfa9c7d445eaaa83f5c603c2bffdda0"}
Nov 03 14:39:39.709986 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.709962    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70ee59143b192d1612ef3b4404d19a020cfa9c7d445eaaa83f5c603c2bffdda0"
Nov 03 14:39:39.710389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.710371    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"
Nov 03 14:39:39.729366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:39.729332    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"]
Nov 03 14:39:40.043307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:40.037296    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:40.270967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:40.270932    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:40.726050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:40.725979    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:41.201138 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.201094    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d" podUID="7c562dae-c343-46b6-b08b-03bf89313f4b" containerName="payload-processing" containerID="cri-o://87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d" gracePeriod=25
Nov 03 14:39:41.386553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.386522    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:41.468272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.468202    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:39:41.495610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.495577    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume\") pod \"7c562dae-c343-46b6-b08b-03bf89313f4b\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") "
Nov 03 14:39:41.495759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.495679    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume\") pod \"7c562dae-c343-46b6-b08b-03bf89313f4b\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") "
Nov 03 14:39:41.495839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.495790    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz\") pod \"7c562dae-c343-46b6-b08b-03bf89313f4b\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") "
Nov 03 14:39:41.495902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.495878    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials\") pod \"7c562dae-c343-46b6-b08b-03bf89313f4b\" (UID: \"7c562dae-c343-46b6-b08b-03bf89313f4b\") "
Nov 03 14:39:41.496034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.496007    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume" pod "7c562dae-c343-46b6-b08b-03bf89313f4b" (UID: "7c562dae-c343-46b6-b08b-03bf89313f4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:39:41.496099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.496073    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume" pod "7c562dae-c343-46b6-b08b-03bf89313f4b" (UID: "7c562dae-c343-46b6-b08b-03bf89313f4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:39:41.496377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.496357    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:41.496471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.496379    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7c562dae-c343-46b6-b08b-03bf89313f4b-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:41.497827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.497806    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials" pod "7c562dae-c343-46b6-b08b-03bf89313f4b" (UID: "7c562dae-c343-46b6-b08b-03bf89313f4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:39:41.498024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.498008    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz" pod "7c562dae-c343-46b6-b08b-03bf89313f4b" (UID: "7c562dae-c343-46b6-b08b-03bf89313f4b"). InnerVolumeSpecName "kube-api-access-lnsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:39:41.597495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.597466    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnsqz\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-kube-api-access-lnsqz\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:41.597495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.597495    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7c562dae-c343-46b6-b08b-03bf89313f4b-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:41.729159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.729079    6481 generic.go:356] "Generic (PLEG): container finished" podID="7c562dae-c343-46b6-b08b-03bf89313f4b" containerID="87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d" exitCode=0
Nov 03 14:39:41.729159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.729125    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"
Nov 03 14:39:41.729159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.729152    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d" event={"ID":"7c562dae-c343-46b6-b08b-03bf89313f4b","Type":"ContainerDied","Data":"87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d"}
Nov 03 14:39:41.729437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.729192    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d" event={"ID":"7c562dae-c343-46b6-b08b-03bf89313f4b","Type":"ContainerDied","Data":"118a36069de9bf1c6dff27e1c8dcb482a48a0f99a1a938ed3d9238a25c5a4b5d"}
Nov 03 14:39:41.729437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.729214    6481 scope.go:122] "RemoveContainer" containerID="87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d"
Nov 03 14:39:41.742991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.742963    6481 scope.go:122] "RemoveContainer" containerID="87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d"
Nov 03 14:39:41.743457 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:39:41.743433    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d\": container with ID starting with 87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d not found: ID does not exist" containerID="87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d"
Nov 03 14:39:41.743569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.743467    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d"} err="failed to get container status \"87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d\": rpc error: code = NotFound desc = could not find container \"87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d\": container with ID starting with 87806ce4929070ca599e6a7f5e6baecd1b0cc5065b8cefc17ea90a99adee5f7d not found: ID does not exist"
Nov 03 14:39:41.786577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.786540    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"]
Nov 03 14:39:41.798855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:41.798823    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bf9485bb6-jrz9d"]
Nov 03 14:39:42.067495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:42.067407    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c562dae-c343-46b6-b08b-03bf89313f4b" path="/var/lib/kubelet/pods/7c562dae-c343-46b6-b08b-03bf89313f4b/volumes"
Nov 03 14:39:42.926090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:42.926058    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:44.044056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.044018    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"]
Nov 03 14:39:44.055120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.055090    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-848a860c-5g6jp"]
Nov 03 14:39:44.066322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.066293    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475cb202-4db4-40ef-8872-8d5a3aa8588a" path="/var/lib/kubelet/pods/475cb202-4db4-40ef-8872-8d5a3aa8588a/volumes"
Nov 03 14:39:44.239882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.239845    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main" containerID="cri-o://6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806" gracePeriod=45
Nov 03 14:39:44.267381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.267334    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main" containerID="cri-o://b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a" gracePeriod=45
Nov 03 14:39:44.539509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.539485    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:39:44.569198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.569144    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:39:44.634973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.634940    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl\" (UniqueName: \"kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl\") pod \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635049    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache\") pod \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635085    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache\") pod \"52ad424d-51fc-400c-bb0e-b7c6defea720\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635118    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm\") pod \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635159    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir\") pod \"52ad424d-51fc-400c-bb0e-b7c6defea720\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635422    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home\") pod \"52ad424d-51fc-400c-bb0e-b7c6defea720\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635481    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs\" (UniqueName: \"kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs\") pod \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635523    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir\") pod \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635553    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache" pod "61f10c56-c20d-4e6f-b0c2-d43c6487eabd" (UID: "61f10c56-c20d-4e6f-b0c2-d43c6487eabd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635611    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm\") pod \"52ad424d-51fc-400c-bb0e-b7c6defea720\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635654    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g\" (UniqueName: \"kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g\") pod \"52ad424d-51fc-400c-bb0e-b7c6defea720\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635699    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home\") pod \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\" (UID: \"61f10c56-c20d-4e6f-b0c2-d43c6487eabd\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635736    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs\") pod \"52ad424d-51fc-400c-bb0e-b7c6defea720\" (UID: \"52ad424d-51fc-400c-bb0e-b7c6defea720\") "
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635768    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home" pod "52ad424d-51fc-400c-bb0e-b7c6defea720" (UID: "52ad424d-51fc-400c-bb0e-b7c6defea720"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.636162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.635930    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir" pod "52ad424d-51fc-400c-bb0e-b7c6defea720" (UID: "52ad424d-51fc-400c-bb0e-b7c6defea720"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636369    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache" pod "52ad424d-51fc-400c-bb0e-b7c6defea720" (UID: "52ad424d-51fc-400c-bb0e-b7c6defea720"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636478    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636495    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636512    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636506    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir" pod "61f10c56-c20d-4e6f-b0c2-d43c6487eabd" (UID: "61f10c56-c20d-4e6f-b0c2-d43c6487eabd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636525    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.637441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.636708    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home" pod "61f10c56-c20d-4e6f-b0c2-d43c6487eabd" (UID: "61f10c56-c20d-4e6f-b0c2-d43c6487eabd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.638910 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.638877    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs" pod "52ad424d-51fc-400c-bb0e-b7c6defea720" (UID: "52ad424d-51fc-400c-bb0e-b7c6defea720"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:39:44.639038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.638935    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl" pod "61f10c56-c20d-4e6f-b0c2-d43c6487eabd" (UID: "61f10c56-c20d-4e6f-b0c2-d43c6487eabd"). InnerVolumeSpecName "kube-api-access-ddbtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:39:44.640016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.639977    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs" pod "61f10c56-c20d-4e6f-b0c2-d43c6487eabd" (UID: "61f10c56-c20d-4e6f-b0c2-d43c6487eabd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:39:44.640703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.640679    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm" pod "52ad424d-51fc-400c-bb0e-b7c6defea720" (UID: "52ad424d-51fc-400c-bb0e-b7c6defea720"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.641489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.641457    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g" pod "52ad424d-51fc-400c-bb0e-b7c6defea720" (UID: "52ad424d-51fc-400c-bb0e-b7c6defea720"). InnerVolumeSpecName "kube-api-access-ctz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:39:44.645568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.645533    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm" pod "61f10c56-c20d-4e6f-b0c2-d43c6487eabd" (UID: "61f10c56-c20d-4e6f-b0c2-d43c6487eabd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:39:44.646102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.646070    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:44.731177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.731139    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerName="payload-processing" containerID="cri-o://79438afb8ff6dfa0b7253a0b079bf4956399966dc8e960159908335b5fda6896" gracePeriod=25
Nov 03 14:39:44.737208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737187    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737208    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737216    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ad424d-51fc-400c-bb0e-b7c6defea720-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737225    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctz5g\" (UniqueName: \"kubernetes.io/projected/52ad424d-51fc-400c-bb0e-b7c6defea720-kube-api-access-ctz5g\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737250    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737265    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ad424d-51fc-400c-bb0e-b7c6defea720-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737274    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddbtl\" (UniqueName: \"kubernetes.io/projected/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-kube-api-access-ddbtl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.737362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.737281    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f10c56-c20d-4e6f-b0c2-d43c6487eabd-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:39:44.749409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.749386    6481 generic.go:356] "Generic (PLEG): container finished" podID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerID="6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806" exitCode=0
Nov 03 14:39:44.749500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.749437    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"
Nov 03 14:39:44.749500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.749468    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" event={"ID":"61f10c56-c20d-4e6f-b0c2-d43c6487eabd","Type":"ContainerDied","Data":"6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806"}
Nov 03 14:39:44.749581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.749513    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb" event={"ID":"61f10c56-c20d-4e6f-b0c2-d43c6487eabd","Type":"ContainerDied","Data":"580a750985443aecbd50bc483a671157b2f0eabf380d3c2b5aa90819a85452fb"}
Nov 03 14:39:44.749581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.749530    6481 scope.go:122] "RemoveContainer" containerID="6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806"
Nov 03 14:39:44.751256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.751217    6481 generic.go:356] "Generic (PLEG): container finished" podID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerID="b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a" exitCode=0
Nov 03 14:39:44.751359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.751277    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" event={"ID":"52ad424d-51fc-400c-bb0e-b7c6defea720","Type":"ContainerDied","Data":"b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a"}
Nov 03 14:39:44.751359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.751283    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"
Nov 03 14:39:44.751359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.751303    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l" event={"ID":"52ad424d-51fc-400c-bb0e-b7c6defea720","Type":"ContainerDied","Data":"5be47fb557293a90343e60ac87067b03b0d8041e83e94a77a869e0f2f039c17e"}
Nov 03 14:39:44.760288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.760267    6481 scope.go:122] "RemoveContainer" containerID="6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806"
Nov 03 14:39:44.760565 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:39:44.760546    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806\": container with ID starting with 6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806 not found: ID does not exist" containerID="6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806"
Nov 03 14:39:44.760634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.760572    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806"} err="failed to get container status \"6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806\": rpc error: code = NotFound desc = could not find container \"6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806\": container with ID starting with 6941142daac7a5517858c9e62283fcbd96eaf51a438acf47f73d5905a2fca806 not found: ID does not exist"
Nov 03 14:39:44.760634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.760596    6481 scope.go:122] "RemoveContainer" containerID="b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a"
Nov 03 14:39:44.773405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.773380    6481 scope.go:122] "RemoveContainer" containerID="b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a"
Nov 03 14:39:44.773834 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:39:44.773811    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a\": container with ID starting with b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a not found: ID does not exist" containerID="b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a"
Nov 03 14:39:44.774031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.774006    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a"} err="failed to get container status \"b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a\": rpc error: code = NotFound desc = could not find container \"b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a\": container with ID starting with b102108a8e5c667c0cfaa023f2e07efa239218b679ba946481755098165d171a not found: ID does not exist"
Nov 03 14:39:44.816257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.816197    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"]
Nov 03 14:39:44.823911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.823838    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/simulator-w3-156ae1-kserve-95c9cb797-nd96l"]
Nov 03 14:39:44.857743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.857717    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"]
Nov 03 14:39:44.864468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.864446    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-156ae1/premium-w3-156ae1-kserve-7688f598b4-8s4bb"]
Nov 03 14:39:44.926661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:44.926622    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:45.176189 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:45.176157    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:45.511732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:45.511651    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:45.543599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:45.543567    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:39:46.066558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:46.066519    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" path="/var/lib/kubelet/pods/52ad424d-51fc-400c-bb0e-b7c6defea720/volumes"
Nov 03 14:39:46.067047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:46.067026    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" path="/var/lib/kubelet/pods/61f10c56-c20d-4e6f-b0c2-d43c6487eabd/volumes"
Nov 03 14:39:47.396589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:47.396553    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:48.043892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:48.043863    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"]
Nov 03 14:39:48.055190 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:48.055167    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251476f1-2dkzx"]
Nov 03 14:39:48.066251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:48.066212    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1af475e-33e4-4521-81a8-c60c3470ed11" path="/var/lib/kubelet/pods/c1af475e-33e4-4521-81a8-c60c3470ed11/volumes"
Nov 03 14:39:48.736951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:48.736920    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:48.964749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:48.964716    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:49.058729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:49.058634    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:39:49.520128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:49.520075    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:51.090220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:51.090187    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:51.339130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:51.338996    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:51.971758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:51.971681    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:52.234307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:52.233254    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:54.312672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:54.312646    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:55.053669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:55.053640    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"]
Nov 03 14:39:55.065006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:55.064981    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2615f60c-vzktg"]
Nov 03 14:39:55.544347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:55.544307    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:39:56.067065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:56.067030    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d" path="/var/lib/kubelet/pods/0b3ffbee-ecd5-4dff-aeb9-32919dd0d70d/volumes"
Nov 03 14:39:56.411686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:56.411654    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:57.904670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:57.904631    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:57.988914 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:39:57.988872    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:39:58.161332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.158949    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:58.633306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.632826    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:58.675264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.674272    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"]
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675583    6481 cpu_manager.go:414] "containerMap: removing container" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675606    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675645    6481 cpu_manager.go:414] "containerMap: removing container" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675654    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675672    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1f9621b6-465d-4b17-8f92-15d830277191" containerName="revoke-keys"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675679    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f9621b6-465d-4b17-8f92-15d830277191" containerName="revoke-keys"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675695    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7c562dae-c343-46b6-b08b-03bf89313f4b" containerName="payload-processing"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675703    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c562dae-c343-46b6-b08b-03bf89313f4b" containerName="payload-processing"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675842    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ad424d-51fc-400c-bb0e-b7c6defea720" containerName="main"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675861    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="61f10c56-c20d-4e6f-b0c2-d43c6487eabd" containerName="main"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675870    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f9621b6-465d-4b17-8f92-15d830277191" containerName="revoke-keys"
Nov 03 14:39:58.677629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.675880    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c562dae-c343-46b6-b08b-03bf89313f4b" containerName="payload-processing"
Nov 03 14:39:58.679854 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.679829    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.694192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.694166    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"]
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.700395    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712642    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712688    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712727    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msppt\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712761    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712791    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712846    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712881    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712919    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.712967    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713005    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713040    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713079    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713122    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713160    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713189    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713264    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltsg\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713298    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.719077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.713326    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.721574 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.721133    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-156ae1-openshift-default-dockercfg-hh5m9\""
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.787798    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"]
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.792857    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.813310    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"]
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.818803    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msppt\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.818850    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.818879    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.818937    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.818968    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819006    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819054    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819092    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819124    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819161    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819204    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819613    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819790    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"]
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819838    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819872    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819933    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltsg\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.819969    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.820398    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.820441    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.820486    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.820520    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.820629    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.820862    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.821197    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.836551    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.837201    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.852045    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.852338    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.880019    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.882375    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.891297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.888704    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:58.893598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.891427    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"]
Nov 03 14:39:58.893598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.892978    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.899425    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltsg\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946457    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946501    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946536    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946607    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbq8l\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946686    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946723    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946809    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946836    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.946889    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.983814    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:58.991098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.989654    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:59.001250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.998066    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:59.001250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:58.998716    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msppt\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:59.006250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.005446    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070307    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbq8l\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070394    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070436    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070508    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070536    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070582    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070645    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070674    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.070711    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.071244    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.084270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.072200    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.087204 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.087173    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.090898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.090826    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.091501 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.091457    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.091679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.091635    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.093284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.092059    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.098384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.093491    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:39:59.099378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.099331    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbq8l\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.120319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.100696    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token\") pod \"e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") " pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.127377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.125479    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:39:59.680475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.680440    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:39:59.860696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.859294    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"]
Nov 03 14:39:59.889519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.889200    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"]
Nov 03 14:39:59.891693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.891356    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.891693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.891437    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.891693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.891484    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.893808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.893774    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.893907 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:39:59.893806    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4070a6_49f1_4bdf_a9a1_c75337bb5bf7.slice/crio-f02f79a8b9640cbf38832711cf2c936804d05ed4e4941acb25c5722715c59323 WatchSource:0}: Error finding container f02f79a8b9640cbf38832711cf2c936804d05ed4e4941acb25c5722715c59323: Status 404 returned error can't find the container with id f02f79a8b9640cbf38832711cf2c936804d05ed4e4941acb25c5722715c59323
Nov 03 14:39:59.893907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.893850    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.894031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.893902    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.896972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.896948    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.897060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.896986    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"]
Nov 03 14:39:59.897060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.897002    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.897060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.897027    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:39:59.989526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.989429    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" event={"ID":"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9","Type":"ContainerStarted","Data":"e27ccf83297d04918f321b5d19b0dbc60927b2879e2a311e0eb11a52533651c1"}
Nov 03 14:39:59.991538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.991440    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" event={"ID":"aca8d9a0-da46-4718-80ca-54f0a25d72f5","Type":"ContainerStarted","Data":"6af75d2eb63936f424fdb9df166d522792114cb6ab9e7fa55ebe4117d649c9e3"}
Nov 03 14:39:59.994017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:39:59.993947    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" event={"ID":"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7","Type":"ContainerStarted","Data":"f02f79a8b9640cbf38832711cf2c936804d05ed4e4941acb25c5722715c59323"}
Nov 03 14:40:00.398252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:00.395120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:00.621714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:00.621678    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:01.054263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.050307    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" event={"ID":"aca8d9a0-da46-4718-80ca-54f0a25d72f5","Type":"ContainerStarted","Data":"2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc"}
Nov 03 14:40:01.059267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.055425    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" event={"ID":"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7","Type":"ContainerStarted","Data":"800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496"}
Nov 03 14:40:01.059267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.058568    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" event={"ID":"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9","Type":"ContainerStarted","Data":"f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf"}
Nov 03 14:40:01.109259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.104297    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:40:01.113266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.110351    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:01.113266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.110441    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:01.129560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.129493    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:40:01.129824 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.129802    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:01.131280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.129958    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:01.132708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.132376    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podStartSLOduration=3.132358583 podStartE2EDuration="3.132358583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:01.118631947 +0000 UTC m=+1679.168663132" watchObservedRunningTime="2026-11-03 14:40:01.132358583 +0000 UTC m=+1679.182389768"
Nov 03 14:40:01.201262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.200135    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podStartSLOduration=3.200116724 podStartE2EDuration="3.200116724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:01.191858806 +0000 UTC m=+1679.241889992" watchObservedRunningTime="2026-11-03 14:40:01.200116724 +0000 UTC m=+1679.250147907"
Nov 03 14:40:01.463721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.463609    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:01.961256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:01.958607    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:02.024254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.024176    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:40:02.025193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.025168    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 14:40:02.025444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.025414    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 14:40:02.094250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.094179    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:02.094766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.094738    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:02.180782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.180728    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:02.180993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:02.180804    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:03.029430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.029394    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 14:40:03.029718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.029691    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 14:40:03.099763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.099725    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:03.100385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.100352    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:03.134971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.134941    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:03.135244 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.135197    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:03.784073 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:03.784032    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:04.035558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:04.034658    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 14:40:04.035558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:04.034727    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 14:40:04.094837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:04.094801    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:04.095027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:04.094852    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:04.130253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:04.129592    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:04.130253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:04.129651    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:05.042441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.020647    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 14:40:05.042441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.020709    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 14:40:05.094559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.094321    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:05.094559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.094379    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:05.159457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.150923    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:05.159457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.150978    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:05.191852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.191693    6481 generic.go:356] "Generic (PLEG): container finished" podID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerID="79438afb8ff6dfa0b7253a0b079bf4956399966dc8e960159908335b5fda6896" exitCode=0
Nov 03 14:40:05.191852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.191826    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" event={"ID":"9a75e5cf-ac0f-491f-98fd-a8e3573e357e","Type":"ContainerDied","Data":"79438afb8ff6dfa0b7253a0b079bf4956399966dc8e960159908335b5fda6896"}
Nov 03 14:40:05.384588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.384503    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:40:05.486253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.485716    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podStartSLOduration=7.4856947609999995 podStartE2EDuration="7.485694761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:01.259851317 +0000 UTC m=+1679.309882501" watchObservedRunningTime="2026-11-03 14:40:05.485694761 +0000 UTC m=+1683.535725947"
Nov 03 14:40:05.497720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.497695    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume\") pod \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") "
Nov 03 14:40:05.497944 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.497931    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs\" (UniqueName: \"kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs\") pod \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\" (UID: \"9a75e5cf-ac0f-491f-98fd-a8e3573e357e\") "
Nov 03 14:40:05.498728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.498704    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume" pod "9a75e5cf-ac0f-491f-98fd-a8e3573e357e" (UID: "9a75e5cf-ac0f-491f-98fd-a8e3573e357e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:40:05.550252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.548413    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs" pod "9a75e5cf-ac0f-491f-98fd-a8e3573e357e" (UID: "9a75e5cf-ac0f-491f-98fd-a8e3573e357e"). InnerVolumeSpecName "kube-api-access-4cgrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:05.599678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.599649    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:05.599678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.599675    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cgrs\" (UniqueName: \"kubernetes.io/projected/9a75e5cf-ac0f-491f-98fd-a8e3573e357e-kube-api-access-4cgrs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:05.952252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:05.951376    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:06.024265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.023509    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 14:40:06.024265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.023575    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 14:40:06.050253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.049701    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"]
Nov 03 14:40:06.083783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.083753    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3260b640-zwq9p"]
Nov 03 14:40:06.094815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.094782    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:06.094928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.094840    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:06.139252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.135128    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:06.139252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.135186    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:06.235303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.217522    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5" event={"ID":"9a75e5cf-ac0f-491f-98fd-a8e3573e357e","Type":"ContainerDied","Data":"c3a62f056577c899ae366a538d95500db6a476352c2da38db8d7e43023247d6c"}
Nov 03 14:40:06.235303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.217615    6481 scope.go:122] "RemoveContainer" containerID="79438afb8ff6dfa0b7253a0b079bf4956399966dc8e960159908335b5fda6896"
Nov 03 14:40:06.235303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.217862    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"
Nov 03 14:40:06.289250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.283783    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"]
Nov 03 14:40:06.291939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.291139    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-156ae1-6d45d5c8f9-g5rf5"]
Nov 03 14:40:06.331282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.331166    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:40:06.657338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.653062    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:06.874258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:06.873679    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:07.020314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.019584    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Nov 03 14:40:07.020314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.019646    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Nov 03 14:40:07.098402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.098364    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Nov 03 14:40:07.098677 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.098652    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Nov 03 14:40:07.147937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.147904    6481 patch_prober.go:29] interesting pod/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body=
Nov 03 14:40:07.148245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.148200    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused"
Nov 03 14:40:07.354808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.328166    6481 scope.go:122] "RemoveContainer" containerID="9cc398736ea8fb45058f6e4495e5b1771180344ebffc2f0c507ce455ec3f4894"
Nov 03 14:40:07.481324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.476849    6481 scope.go:122] "RemoveContainer" containerID="74a2070fb6379d4216fc358878feb63989eb83a4a0d628e6b97089c3e52dc1cb"
Nov 03 14:40:07.501510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.501125    6481 scope.go:122] "RemoveContainer" containerID="306d630ed6801d84088fede03ad1c5b6db3be787cddb78ee4f8e1bd1c7f9115c"
Nov 03 14:40:07.519306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.518975    6481 scope.go:122] "RemoveContainer" containerID="ec6a3c2680bd7b4df49712dbfa3d6f1df2a3a373778322acd3a14b68c3a965ee"
Nov 03 14:40:07.850579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:07.850500    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:08.108639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.108602    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:08.113117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.113088    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b" path="/var/lib/kubelet/pods/6a5ba7b8-7eb7-43a0-85bb-419cf4c3ec0b/volumes"
Nov 03 14:40:08.113940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.113916    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" path="/var/lib/kubelet/pods/9a75e5cf-ac0f-491f-98fd-a8e3573e357e/volumes"
Nov 03 14:40:08.341161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.341129    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:08.371894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.370415    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:40:08.371894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.371780    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:40:08.374160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.374105    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:40:08.408032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.407108    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:40:08.408355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.408317    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:40:08.410952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.410933    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:40:08.498326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.498298    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:40:08.499216 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.499111    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:40:08.520413 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:08.520374    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:40:09.290255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:09.285817    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:09.520253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:09.516633    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:09.752262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:09.750801    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:10.059312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:10.055713    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:10.494130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:10.494098    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:10.769076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:10.768304    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:11.020611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.020454    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:11.459840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.459702    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:11.773369 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.773294    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"]
Nov 03 14:40:11.773820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.773505    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy" containerID="cri-o://f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf" gracePeriod=30
Nov 03 14:40:11.783335 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.783311    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"]
Nov 03 14:40:11.783498 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.783474    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy" containerID="cri-o://2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc" gracePeriod=30
Nov 03 14:40:11.792618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.792593    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"]
Nov 03 14:40:11.792755 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.792738    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy" containerID="cri-o://800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496" gracePeriod=30
Nov 03 14:40:11.921151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:11.921115    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:12.354743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:12.354713    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:13.187410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:13.187365    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:14.202582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:14.202551    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:40:15.195946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:15.195917    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:17.047582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.047523    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:40:17.082130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.082108    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:40:17.090591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.090573    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:40:17.142004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.141977    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142033    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142064    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142083    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142131    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142152    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142217    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142282    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142315    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142363    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142405    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142437    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142467    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142507    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142554    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142613    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo\") pod \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\" (UID: \"aca8d9a0-da46-4718-80ca-54f0a25d72f5\") "
Nov 03 14:40:17.142889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142670    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.142889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142798    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.143173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142738    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data\") pod \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\" (UID: \"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9\") "
Nov 03 14:40:17.143173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.142912    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.143329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143256    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:40:17.143329 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143319    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:40:17.143543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143520    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.143602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143547    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.143602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143561    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.143602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143580    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.143862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.143841    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.144333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144286    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.144425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144371    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.144608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144532    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.144608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144587    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.144860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144789    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.144860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144817    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "kube-api-access-gltsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:17.144860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.144841    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.146560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.146540    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:40:17.146717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.146701    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "kube-api-access-msppt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:17.146717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.146703    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:40:17.146830 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.146705    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.146870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.146833    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token" pod "aca8d9a0-da46-4718-80ca-54f0a25d72f5" (UID: "aca8d9a0-da46-4718-80ca-54f0a25d72f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:17.148351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.148330    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token" pod "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" (UID: "70fdf8ff-6444-4e8b-83d8-5ea1163f78e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.253418    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.253518    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.253587    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.253636    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.253682    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.254034    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.254079    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.254111    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.254197    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data\") pod \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\" (UID: \"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7\") "
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.254966    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.254991    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255005    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255019    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255036    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255048    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255061    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255074    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gltsg\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-kube-api-access-gltsg\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255089    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255144    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255163    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255176    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msppt\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-kube-api-access-msppt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255189    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aca8d9a0-da46-4718-80ca-54f0a25d72f5-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255205    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.255538    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.256080    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.256362    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.256712 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.256686    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.259173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.256964    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.267269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.263386    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:40:17.267269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.263604    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "kube-api-access-zbq8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:17.267269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.264402    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:40:17.267269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.266844    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo" pod "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" (UID: "4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:40:17.322290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.322198    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:17.356355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356327    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356350    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356361    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356370    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbq8l\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-kube-api-access-zbq8l\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356380    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356388    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356395    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356403    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.356560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.356410    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:40:17.440815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.440783    6481 generic.go:356] "Generic (PLEG): container finished" podID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerID="2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc" exitCode=0
Nov 03 14:40:17.441024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.440839    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"
Nov 03 14:40:17.441024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.440871    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" event={"ID":"aca8d9a0-da46-4718-80ca-54f0a25d72f5","Type":"ContainerDied","Data":"2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc"}
Nov 03 14:40:17.441024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.440936    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh" event={"ID":"aca8d9a0-da46-4718-80ca-54f0a25d72f5","Type":"ContainerDied","Data":"6af75d2eb63936f424fdb9df166d522792114cb6ab9e7fa55ebe4117d649c9e3"}
Nov 03 14:40:17.441024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.440969    6481 scope.go:122] "RemoveContainer" containerID="2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc"
Nov 03 14:40:17.442316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.442298    6481 generic.go:356] "Generic (PLEG): container finished" podID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerID="800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496" exitCode=0
Nov 03 14:40:17.442419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.442404    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" event={"ID":"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7","Type":"ContainerDied","Data":"800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496"}
Nov 03 14:40:17.442489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.442442    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh" event={"ID":"4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7","Type":"ContainerDied","Data":"f02f79a8b9640cbf38832711cf2c936804d05ed4e4941acb25c5722715c59323"}
Nov 03 14:40:17.442547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.442495    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"
Nov 03 14:40:17.446943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.446920    6481 generic.go:356] "Generic (PLEG): container finished" podID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerID="f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf" exitCode=0
Nov 03 14:40:17.447076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.446965    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" event={"ID":"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9","Type":"ContainerDied","Data":"f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf"}
Nov 03 14:40:17.447076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.447004    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z" event={"ID":"70fdf8ff-6444-4e8b-83d8-5ea1163f78e9","Type":"ContainerDied","Data":"e27ccf83297d04918f321b5d19b0dbc60927b2879e2a311e0eb11a52533651c1"}
Nov 03 14:40:17.447076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.447028    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"
Nov 03 14:40:17.452504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.452483    6481 scope.go:122] "RemoveContainer" containerID="2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc"
Nov 03 14:40:17.452809 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:40:17.452792    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc\": container with ID starting with 2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc not found: ID does not exist" containerID="2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc"
Nov 03 14:40:17.452896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.452819    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc"} err="failed to get container status \"2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc\": rpc error: code = NotFound desc = could not find container \"2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc\": container with ID starting with 2979092080d2e4d10ce1e8ba7d34552d9b737692c152a27f789731e22f7d04cc not found: ID does not exist"
Nov 03 14:40:17.452896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.452843    6481 scope.go:122] "RemoveContainer" containerID="800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496"
Nov 03 14:40:17.462280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.462260    6481 scope.go:122] "RemoveContainer" containerID="800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496"
Nov 03 14:40:17.462533 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:40:17.462512    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496\": container with ID starting with 800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496 not found: ID does not exist" containerID="800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496"
Nov 03 14:40:17.462616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.462543    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496"} err="failed to get container status \"800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496\": rpc error: code = NotFound desc = could not find container \"800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496\": container with ID starting with 800693efd1d1048b248141a30079af428a71409fb3d67f75ef7cfa8ab4ffe496 not found: ID does not exist"
Nov 03 14:40:17.462616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.462564    6481 scope.go:122] "RemoveContainer" containerID="f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf"
Nov 03 14:40:17.471043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.471023    6481 scope.go:122] "RemoveContainer" containerID="f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf"
Nov 03 14:40:17.471338 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:40:17.471311    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf\": container with ID starting with f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf not found: ID does not exist" containerID="f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf"
Nov 03 14:40:17.471435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.471349    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf"} err="failed to get container status \"f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf\": rpc error: code = NotFound desc = could not find container \"f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf\": container with ID starting with f5e30204ecb3dd2f68f597217b52155b8f6b9076eed0ed0e801b2d97bb328ebf not found: ID does not exist"
Nov 03 14:40:17.495643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.495617    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"]
Nov 03 14:40:17.508120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.508100    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-fjmmh"]
Nov 03 14:40:17.535511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.535487    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"]
Nov 03 14:40:17.541421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.541401    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-djlxh"]
Nov 03 14:40:17.592186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.592114    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"]
Nov 03 14:40:17.595704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:17.595673    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-156ae1-openshift-default-859f8b99df-lnc6z"]
Nov 03 14:40:18.064078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:18.064050    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:18.066437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:18.066416    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" path="/var/lib/kubelet/pods/4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7/volumes"
Nov 03 14:40:18.066871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:18.066858    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" path="/var/lib/kubelet/pods/70fdf8ff-6444-4e8b-83d8-5ea1163f78e9/volumes"
Nov 03 14:40:18.067269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:18.067257    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" path="/var/lib/kubelet/pods/aca8d9a0-da46-4718-80ca-54f0a25d72f5/volumes"
Nov 03 14:40:18.292692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:18.292662    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:19.403905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:19.403870    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:21.420766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:21.420737    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:23.534126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:23.534091    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:25.604257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:25.604212    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:27.660036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:27.660006    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:27.985436 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:40:27.985357    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:40:28.579334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:28.579306    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:40:29.706067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:29.706034    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:31.713656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:31.713619    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:33.714621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:33.714592    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:33.938709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:33.938677    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:35.255741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:35.255706    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:35.744112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:35.744079    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:36.179253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:36.178696    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:40:37.812655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:37.812628    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:39.840972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:39.840940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:41.041921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:41.041883    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"]
Nov 03 14:40:41.053285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:41.053257    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61874855-f5km2"]
Nov 03 14:40:41.853245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:41.853208    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:42.066487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:42.066454    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df9033c-d425-4a69-904a-91ecf27c711f" path="/var/lib/kubelet/pods/7df9033c-d425-4a69-904a-91ecf27c711f/volumes"
Nov 03 14:40:43.822545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:43.822468    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:45.863186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:45.863153    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:47.831214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:47.831182    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:49.914529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:49.914497    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:51.613901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:51.613861    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:40:51.912670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:51.912636    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:53.928977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:53.928943    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:56.007606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:56.007571    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:57.986248 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:40:57.986190    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:40:57.999808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:57.999783    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:59.776802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.776772    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:40:59.974205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974166    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"]
Nov 03 14:40:59.974852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974832    6481 cpu_manager.go:414] "containerMap: removing container" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy"
Nov 03 14:40:59.974905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974854    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy"
Nov 03 14:40:59.974905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974870    6481 cpu_manager.go:414] "containerMap: removing container" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy"
Nov 03 14:40:59.974905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974875    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy"
Nov 03 14:40:59.974905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974883    6481 cpu_manager.go:414] "containerMap: removing container" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy"
Nov 03 14:40:59.974905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974888    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy"
Nov 03 14:40:59.975076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974924    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerName="payload-processing"
Nov 03 14:40:59.975076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974930    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerName="payload-processing"
Nov 03 14:40:59.975076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.974998    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a75e5cf-ac0f-491f-98fd-a8e3573e357e" containerName="payload-processing"
Nov 03 14:40:59.975076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.975007    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="aca8d9a0-da46-4718-80ca-54f0a25d72f5" containerName="istio-proxy"
Nov 03 14:40:59.975076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.975014    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="70fdf8ff-6444-4e8b-83d8-5ea1163f78e9" containerName="istio-proxy"
Nov 03 14:40:59.975076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.975020    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e4070a6-49f1-4bdf-a9a1-c75337bb5bf7" containerName="istio-proxy"
Nov 03 14:40:59.977717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.977701    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:40:59.999253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:40:59.999210    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"]
Nov 03 14:41:00.006880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.006855    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:41:00.007080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.006863    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:41:00.007198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.006876    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:41:00.007198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.006955    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:41:00.038086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.038012    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:00.093066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.093039    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd9k6\" (UniqueName: \"kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.093195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.093084    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.093275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.093257    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.194527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.194488    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.194664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.194573    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd9k6\" (UniqueName: \"kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.194664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.194610    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.195080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.195058    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.196978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.196960    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.215745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.215720    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd9k6\" (UniqueName: \"kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6\") pod \"maas-api-revoke-sub-9bfe7505-d6dpz\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.291461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.291401    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:00.592371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.592338    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"]
Nov 03 14:41:00.593647 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:41:00.593619    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70fa97c_d6f1_444c_b8cd_8da49053182a.slice/crio-5afa5e9bf8f1dd356051961131eb7a83c06506e9bbf87b584961cf9a9008eaf1 WatchSource:0}: Error finding container 5afa5e9bf8f1dd356051961131eb7a83c06506e9bbf87b584961cf9a9008eaf1: Status 404 returned error can't find the container with id 5afa5e9bf8f1dd356051961131eb7a83c06506e9bbf87b584961cf9a9008eaf1
Nov 03 14:41:00.712330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:00.712301    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz" event={"ID":"e70fa97c-d6f1-444c-b8cd-8da49053182a","Type":"ContainerStarted","Data":"5afa5e9bf8f1dd356051961131eb7a83c06506e9bbf87b584961cf9a9008eaf1"}
Nov 03 14:41:01.719963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:01.719868    6481 generic.go:356] "Generic (PLEG): container finished" podID="e70fa97c-d6f1-444c-b8cd-8da49053182a" containerID="c7c46201198dfc12aee4d12d6eee1ca2190dc878808957dc7c4720610f9ce9c3" exitCode=0
Nov 03 14:41:01.719963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:01.719926    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz" event={"ID":"e70fa97c-d6f1-444c-b8cd-8da49053182a","Type":"ContainerDied","Data":"c7c46201198dfc12aee4d12d6eee1ca2190dc878808957dc7c4720610f9ce9c3"}
Nov 03 14:41:02.015350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:02.015256    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:02.861150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:02.861128    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:03.022787 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.022704    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca\") pod \"e70fa97c-d6f1-444c-b8cd-8da49053182a\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") "
Nov 03 14:41:03.022957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.022855    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca\") pod \"e70fa97c-d6f1-444c-b8cd-8da49053182a\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") "
Nov 03 14:41:03.022957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.022925    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6\" (UniqueName: \"kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6\") pod \"e70fa97c-d6f1-444c-b8cd-8da49053182a\" (UID: \"e70fa97c-d6f1-444c-b8cd-8da49053182a\") "
Nov 03 14:41:03.023187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.023163    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca" pod "e70fa97c-d6f1-444c-b8cd-8da49053182a" (UID: "e70fa97c-d6f1-444c-b8cd-8da49053182a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:41:03.023370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.023356    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:03.024815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.024790    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca" pod "e70fa97c-d6f1-444c-b8cd-8da49053182a" (UID: "e70fa97c-d6f1-444c-b8cd-8da49053182a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:41:03.024954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.024930    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6" pod "e70fa97c-d6f1-444c-b8cd-8da49053182a" (UID: "e70fa97c-d6f1-444c-b8cd-8da49053182a"). InnerVolumeSpecName "kube-api-access-rd9k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:41:03.124416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.124388    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd9k6\" (UniqueName: \"kubernetes.io/projected/e70fa97c-d6f1-444c-b8cd-8da49053182a-kube-api-access-rd9k6\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:03.124416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.124412    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e70fa97c-d6f1-444c-b8cd-8da49053182a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:03.733527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.733486    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz" event={"ID":"e70fa97c-d6f1-444c-b8cd-8da49053182a","Type":"ContainerDied","Data":"5afa5e9bf8f1dd356051961131eb7a83c06506e9bbf87b584961cf9a9008eaf1"}
Nov 03 14:41:03.733825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.733543    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5afa5e9bf8f1dd356051961131eb7a83c06506e9bbf87b584961cf9a9008eaf1"
Nov 03 14:41:03.733825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:03.733509    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"
Nov 03 14:41:04.018345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:04.018225    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:05.929453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:05.929421    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:07.622241 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:07.622202    6481 scope.go:122] "RemoveContainer" containerID="2abf7b5fcfec1cd0e76c7890e93d561fe21ebbcc9bc3dc7a9eee4e520069be72"
Nov 03 14:41:07.634120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:07.634093    6481 scope.go:122] "RemoveContainer" containerID="a39e5a793196baf861c7469eb2f6be99e3c0eb4fa04485ee26ea511c2bb2d2e9"
Nov 03 14:41:07.642516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:07.642497    6481 scope.go:122] "RemoveContainer" containerID="41f2d78fff7655197c4da08ba567ba047c78c057296f663d4cba1ca02e7a2e3e"
Nov 03 14:41:07.650794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:07.650779    6481 scope.go:122] "RemoveContainer" containerID="6c5e9f5a8a035fa001e3eebe4936b8c28ff439da3180b7962f9ec949ef681b2a"
Nov 03 14:41:07.658525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:07.658508    6481 scope.go:122] "RemoveContainer" containerID="abd966aa1461c31ed9b19e2244a2886cf70b13f1bde5dcb94501bda7a0a4d5e7"
Nov 03 14:41:08.032289 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:08.032219    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:10.029278 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:10.029246    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:11.611891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:11.611857    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:11.838440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:11.838372    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:12.059470 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.059436    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:12.168290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.168259    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"]
Nov 03 14:41:12.168746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.168730    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e70fa97c-d6f1-444c-b8cd-8da49053182a" containerName="revoke-keys"
Nov 03 14:41:12.168791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.168749    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70fa97c-d6f1-444c-b8cd-8da49053182a" containerName="revoke-keys"
Nov 03 14:41:12.168859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.168849    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70fa97c-d6f1-444c-b8cd-8da49053182a" containerName="revoke-keys"
Nov 03 14:41:12.173997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.173980    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.194494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.194465    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"]
Nov 03 14:41:12.202656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.202630    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:41:12.202760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.202724    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:41:12.202816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.202800    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:41:12.202889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.202871    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:41:12.213380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.213358    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.213559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.213541    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968nd\" (UniqueName: \"kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.213647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.213606    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.314465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.314401    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968nd\" (UniqueName: \"kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.314465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.314444    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.314637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.314468    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.315090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.315067    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.316740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.316720    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.334541 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.334516    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-968nd\" (UniqueName: \"kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd\") pod \"maas-api-revoke-sub-0db7c9cd-n96kz\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.486780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.486751    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:12.786049 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:12.786018    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"]
Nov 03 14:41:12.787172 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:41:12.787146    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905f76c8_5ac6_43ac_b9ca_05d663fd31c6.slice/crio-aaac59bfff2beb1b99f6edd8a7d717389730745a6570019372ef1d5946e12f1b WatchSource:0}: Error finding container aaac59bfff2beb1b99f6edd8a7d717389730745a6570019372ef1d5946e12f1b: Status 404 returned error can't find the container with id aaac59bfff2beb1b99f6edd8a7d717389730745a6570019372ef1d5946e12f1b
Nov 03 14:41:13.752038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:13.752007    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:41:13.794146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:13.794114    6481 generic.go:356] "Generic (PLEG): container finished" podID="905f76c8-5ac6-43ac-b9ca-05d663fd31c6" containerID="76000b377f60950e320dda180b8cc4f5b110401988a77cfcd5ef2f836c3df43f" exitCode=0
Nov 03 14:41:13.794512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:13.794204    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz" event={"ID":"905f76c8-5ac6-43ac-b9ca-05d663fd31c6","Type":"ContainerDied","Data":"76000b377f60950e320dda180b8cc4f5b110401988a77cfcd5ef2f836c3df43f"}
Nov 03 14:41:13.794512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:13.794281    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz" event={"ID":"905f76c8-5ac6-43ac-b9ca-05d663fd31c6","Type":"ContainerStarted","Data":"aaac59bfff2beb1b99f6edd8a7d717389730745a6570019372ef1d5946e12f1b"}
Nov 03 14:41:14.116472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:14.116443    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:14.956833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:14.956804    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:15.045012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.044976    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd\" (UniqueName: \"kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd\") pod \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") "
Nov 03 14:41:15.045223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.045126    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca\") pod \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") "
Nov 03 14:41:15.045608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.045585    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca\") pod \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\" (UID: \"905f76c8-5ac6-43ac-b9ca-05d663fd31c6\") "
Nov 03 14:41:15.046194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.046163    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca" pod "905f76c8-5ac6-43ac-b9ca-05d663fd31c6" (UID: "905f76c8-5ac6-43ac-b9ca-05d663fd31c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:41:15.047418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.047387    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd" pod "905f76c8-5ac6-43ac-b9ca-05d663fd31c6" (UID: "905f76c8-5ac6-43ac-b9ca-05d663fd31c6"). InnerVolumeSpecName "kube-api-access-968nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:41:15.047634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.047609    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca" pod "905f76c8-5ac6-43ac-b9ca-05d663fd31c6" (UID: "905f76c8-5ac6-43ac-b9ca-05d663fd31c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:41:15.146795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.146754    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968nd\" (UniqueName: \"kubernetes.io/projected/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-kube-api-access-968nd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:15.146795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.146784    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:15.146795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.146794    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905f76c8-5ac6-43ac-b9ca-05d663fd31c6-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:15.807394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.807365    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"
Nov 03 14:41:15.807570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.807396    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz" event={"ID":"905f76c8-5ac6-43ac-b9ca-05d663fd31c6","Type":"ContainerDied","Data":"aaac59bfff2beb1b99f6edd8a7d717389730745a6570019372ef1d5946e12f1b"}
Nov 03 14:41:15.807570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:15.807462    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaac59bfff2beb1b99f6edd8a7d717389730745a6570019372ef1d5946e12f1b"
Nov 03 14:41:16.168191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:16.168162    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:18.114009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:18.113981    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:20.116013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:20.115981    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:22.020703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:22.020672    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:23.701184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:23.701103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:23.924982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:23.924946    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:24.158333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:24.158305    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:24.388645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:24.388611    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:25.883790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:25.883755    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:41:26.098611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:26.098578    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:27.985391 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:41:27.985354    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:41:28.105940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:28.105909    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:30.120415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:30.120383    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:31.967595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:31.967565    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:32.203392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:32.203359    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:32.436925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:32.436897    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:32.747048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:32.746980    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:33.405815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:33.405778    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:33.635280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:33.634336    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:34.211110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:34.211080    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:34.913377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:34.913333    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:35.811032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:35.810993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:35.814656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:35.814455    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:41:36.268253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:36.261090    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:38.217046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:38.217015    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:40.208047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:40.208011    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:42.210379 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:42.210348    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:44.210814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:44.210781    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:46.195294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:46.195238    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:46.449082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:46.448991    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:48.310097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:48.310067    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:49.835992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:49.835958    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"]
Nov 03 14:41:49.836498 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:49.836211    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy" containerID="cri-o://b7e920cfd75f6a224775f3a114b906acb17c49ae1979b3a12fde6c5c4273b4b2" gracePeriod=30
Nov 03 14:41:50.407021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:50.406985    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:52.528045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:52.527995    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:53.153530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:53.153497    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:41:54.492907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:54.492874    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:55.057658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.057627    6481 generic.go:356] "Generic (PLEG): container finished" podID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerID="b7e920cfd75f6a224775f3a114b906acb17c49ae1979b3a12fde6c5c4273b4b2" exitCode=0
Nov 03 14:41:55.057823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.057664    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" event={"ID":"2378d404-dfd6-420c-8455-a31b8dbb7847","Type":"ContainerDied","Data":"b7e920cfd75f6a224775f3a114b906acb17c49ae1979b3a12fde6c5c4273b4b2"}
Nov 03 14:41:55.099880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.099855    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:41:55.122200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122149    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122200 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122192    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122209    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122275    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122326    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122365    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122413    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122491    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122578    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket\") pod \"2378d404-dfd6-420c-8455-a31b8dbb7847\" (UID: \"2378d404-dfd6-420c-8455-a31b8dbb7847\") "
Nov 03 14:41:55.122773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122581    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:41:55.122773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122716    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:41:55.122886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122781    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:41:55.122886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122837    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:41:55.123003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.122906    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:41:55.123206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.123188    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.123308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.123210    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2378d404-dfd6-420c-8455-a31b8dbb7847-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.123308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.123222    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.123308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.123259    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.123308 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.123271    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.124469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.124441    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:41:55.124559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.124478    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:41:55.124786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.124770    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "kube-api-access-kqsj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:41:55.125149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.125131    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy" pod "2378d404-dfd6-420c-8455-a31b8dbb7847" (UID: "2378d404-dfd6-420c-8455-a31b8dbb7847"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:41:55.224419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.224391    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqsj5\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-kube-api-access-kqsj5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.224419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.224418    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.224597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.224427    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:55.224597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:55.224437    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2378d404-dfd6-420c-8455-a31b8dbb7847-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:41:56.065714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:56.065685    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"
Nov 03 14:41:56.066730 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:56.066701    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl" event={"ID":"2378d404-dfd6-420c-8455-a31b8dbb7847","Type":"ContainerDied","Data":"f0a81e78bebd13898e3b18a029a11e19aaf837641dd42990bb696fba34f8f397"}
Nov 03 14:41:56.066876 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:56.066756    6481 scope.go:122] "RemoveContainer" containerID="b7e920cfd75f6a224775f3a114b906acb17c49ae1979b3a12fde6c5c4273b4b2"
Nov 03 14:41:56.132105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:56.132082    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"]
Nov 03 14:41:56.144907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:56.143776    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-brttl"]
Nov 03 14:41:56.518960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:56.518928    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:57.986001 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:41:57.985964    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:41:58.067075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:58.067032    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" path="/var/lib/kubelet/pods/2378d404-dfd6-420c-8455-a31b8dbb7847/volumes"
Nov 03 14:41:58.511594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:58.511555    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:41:59.902331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:41:59.902263    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:00.498686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:00.498656    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:02.538036 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:02.538003    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:04.515707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:04.515670    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:06.516166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:06.516133    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:07.706676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:07.706639    6481 scope.go:122] "RemoveContainer" containerID="1c8ca0063fe228ed6eff218dc1c3b6009bce5fe7eef0f6dff4a0656c496b322b"
Nov 03 14:42:08.510121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:08.510092    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:10.351487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:10.351455    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:42:10.512881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:10.512844    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:12.512655 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:12.512620    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:14.526304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:14.526219    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:15.940183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:15.940153    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:16.177397 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:16.177360    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:16.600293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:16.599308    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:18.551270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:18.549811    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:42:18.615313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:18.615245    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:20.618696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:20.618663    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:22.646351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:22.646289    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:24.607368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:24.607336    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:25.041513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:25.041484    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"]
Nov 03 14:42:25.052855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:25.052827    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b47599cc-76btz"]
Nov 03 14:42:26.066217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:26.066179    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525546a1-64b1-410b-9d0f-847799da8e90" path="/var/lib/kubelet/pods/525546a1-64b1-410b-9d0f-847799da8e90/volumes"
Nov 03 14:42:26.672009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:26.671966    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:27.985309 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:42:27.985276    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:42:28.644566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:28.644522    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:30.620282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:30.620218    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:32.614137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:32.614103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:34.500499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:34.500469    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:36.420620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:36.420584    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:37.039709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:37.039673    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"]
Nov 03 14:42:37.050902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:37.050871    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eac3785-zbbxb"]
Nov 03 14:42:38.045222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:38.045188    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"]
Nov 03 14:42:38.056641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:38.056616    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a67309-k8k8l"]
Nov 03 14:42:38.065591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:38.065568    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db33e163-d7ca-4dae-81cd-622e0c446d6f" path="/var/lib/kubelet/pods/db33e163-d7ca-4dae-81cd-622e0c446d6f/volumes"
Nov 03 14:42:38.066004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:38.065990    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4418f37-2100-491d-b101-4839d94d6f43" path="/var/lib/kubelet/pods/e4418f37-2100-491d-b101-4839d94d6f43/volumes"
Nov 03 14:42:38.407721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:38.407685    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:39.896616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:39.896583    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:42:40.404791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:40.404758    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:42.314997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:42.314964    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:44.304412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:44.304382    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:46.309149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:46.309115    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:48.306099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:48.306063    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:50.042042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:50.042010    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"]
Nov 03 14:42:50.052809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:50.052783    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c239883d-9mvxf"]
Nov 03 14:42:50.066177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:50.066152    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5135900c-d06b-4479-90c3-3468fdfb5397" path="/var/lib/kubelet/pods/5135900c-d06b-4479-90c3-3468fdfb5397/volumes"
Nov 03 14:42:50.329565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:50.329484    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:51.698473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:51.698439    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:42:52.371332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:52.371296    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:54.326237 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:54.326191    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:56.322774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:56.322739    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:57.843955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:57.843919    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:58.004493 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:42:58.004453    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:42:58.107960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:58.107853    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:42:58.412140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:42:58.412089    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:00.034508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:00.034473    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:00.370019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:00.369928    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:43:00.431425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:00.431389    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:02.039853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:02.039817    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"]
Nov 03 14:43:02.051101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:02.051072    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a89f400-4bwbm"]
Nov 03 14:43:02.066281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:02.066252    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445bc7c6-b5d1-4060-94da-77c8ec523f7b" path="/var/lib/kubelet/pods/445bc7c6-b5d1-4060-94da-77c8ec523f7b/volumes"
Nov 03 14:43:02.516903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:02.516864    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:04.528017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:04.527976    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:04.767277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:04.767208    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:05.013348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:05.013308    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:06.634765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:06.634708    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:08.279811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:08.279779    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:08.633137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:08.633032    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:10.525078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:10.525028    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:11.102611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:11.102583    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:11.326613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:11.326584    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:12.509565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:12.509537    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:13.346094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346057    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"]
Nov 03 14:43:13.346551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346537    6481 cpu_manager.go:414] "containerMap: removing container" podUID="905f76c8-5ac6-43ac-b9ca-05d663fd31c6" containerName="revoke-keys"
Nov 03 14:43:13.346619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346551    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905f76c8-5ac6-43ac-b9ca-05d663fd31c6" containerName="revoke-keys"
Nov 03 14:43:13.346619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346565    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy"
Nov 03 14:43:13.346619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346570    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy"
Nov 03 14:43:13.346717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346639    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="905f76c8-5ac6-43ac-b9ca-05d663fd31c6" containerName="revoke-keys"
Nov 03 14:43:13.346717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.346650    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2378d404-dfd6-420c-8455-a31b8dbb7847" containerName="istio-proxy"
Nov 03 14:43:13.351285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.351253    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.370075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.370044    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-084a61fd1674\""
Nov 03 14:43:13.371849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.371827    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"]
Nov 03 14:43:13.380884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.380856    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-5b62df50d3b5\""
Nov 03 14:43:13.381012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.380886    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-331c9e\""
Nov 03 14:43:13.381012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.380907    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-2720f8e2922f\""
Nov 03 14:43:13.381012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.380913    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"openshift-service-ca.crt\""
Nov 03 14:43:13.381012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.380935    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"agc-cred-b0be02ea2a5b\""
Nov 03 14:43:13.381012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.380973    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"routing-overlay\""
Nov 03 14:43:13.381248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.381062    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"payload-processing-external-model-e2e--e9e6f325-dockercfg-nzhch\""
Nov 03 14:43:13.381248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.381169    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-331c9e\"/\"kube-root-ca.crt\""
Nov 03 14:43:13.465402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.465365    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.465402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.465407    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7wh\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.465604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.465491    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.465604 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.465558    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.566994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.566960    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.566994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.567001    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7wh\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.567480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.567130    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.567480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.567222    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.567804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.567779    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.568002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.567981    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.569667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.569643    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.589545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.589519    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7wh\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-946748crm\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.664321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.664301    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:13.893276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.893211    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:43:13.984568 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:43:13.984515    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab6ed9b_4aeb_4e61_b15f_c27b642fc243.slice/crio-f1e3b9dfaa57b7993e29c5df12dcba52ea3c5627fcfb36951d081002373febc1 WatchSource:0}: Error finding container f1e3b9dfaa57b7993e29c5df12dcba52ea3c5627fcfb36951d081002373febc1: Status 404 returned error can't find the container with id f1e3b9dfaa57b7993e29c5df12dcba52ea3c5627fcfb36951d081002373febc1
Nov 03 14:43:13.987756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:13.987356    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"]
Nov 03 14:43:14.546422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:14.545262    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" event={"ID":"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243","Type":"ContainerStarted","Data":"de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e"}
Nov 03 14:43:14.546422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:14.545370    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" event={"ID":"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243","Type":"ContainerStarted","Data":"f1e3b9dfaa57b7993e29c5df12dcba52ea3c5627fcfb36951d081002373febc1"}
Nov 03 14:43:14.546422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:14.546365    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:14.604255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:14.601850    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" podStartSLOduration=1.60183049 podStartE2EDuration="1.60183049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:43:14.593505176 +0000 UTC m=+1872.643536359" watchObservedRunningTime="2026-11-03 14:43:14.60183049 +0000 UTC m=+1872.651861675"
Nov 03 14:43:14.655791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:14.655755    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:15.042275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:15.042246    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"]
Nov 03 14:43:15.046337 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:15.046316    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54af72fa-579td"]
Nov 03 14:43:16.066749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:16.066717    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f9835a0-016e-46e1-a9d1-398b98635eb7" path="/var/lib/kubelet/pods/2f9835a0-016e-46e1-a9d1-398b98635eb7/volumes"
Nov 03 14:43:16.710108 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:16.710077    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:18.715327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:18.715298    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:20.715444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:20.715412    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:22.709139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:22.709107    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:24.738572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:24.738538    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:26.559488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:26.559461    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:43:26.699018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:26.698983    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:27.985022 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:43:27.984987    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:43:28.740151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:28.740120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:29.039617 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:29.039536    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"]
Nov 03 14:43:29.050780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:29.050757    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf274bc-6ct5g"]
Nov 03 14:43:30.065748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:30.065714    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ec9411-5fb7-4467-afc8-e646daa8305c" path="/var/lib/kubelet/pods/f1ec9411-5fb7-4467-afc8-e646daa8305c/volumes"
Nov 03 14:43:30.728416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:30.728379    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:32.819011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:32.818979    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:34.808902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:34.808874    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:35.179738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:35.179706    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:43:36.838844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:36.838812    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:38.923966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:38.923929    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:40.931971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:40.931942    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:43.018174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:43.018103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:45.021768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:45.021709    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:46.995205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:46.995170    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:47.185380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.185351    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:43:47.605484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.605443    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"]
Nov 03 14:43:47.605739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.605691    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j" podUID="a25d4e45-29be-4581-b786-68e95371cde6" containerName="discovery" containerID="cri-o://da721857c7598744f60fef2bcfb50aede42d1210a808c374a54b019d17800cd5" gracePeriod=30
Nov 03 14:43:47.739577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.739531    6481 generic.go:356] "Generic (PLEG): container finished" podID="a25d4e45-29be-4581-b786-68e95371cde6" containerID="da721857c7598744f60fef2bcfb50aede42d1210a808c374a54b019d17800cd5" exitCode=0
Nov 03 14:43:47.739748 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.739594    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j" event={"ID":"a25d4e45-29be-4581-b786-68e95371cde6","Type":"ContainerDied","Data":"da721857c7598744f60fef2bcfb50aede42d1210a808c374a54b019d17800cd5"}
Nov 03 14:43:47.876784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.876753    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:43:47.938120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938086    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938137    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938169    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938198    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938222    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938258    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938448    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts\") pod \"a25d4e45-29be-4581-b786-68e95371cde6\" (UID: \"a25d4e45-29be-4581-b786-68e95371cde6\") "
Nov 03 14:43:47.938791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938749    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:43:47.938915 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.938895    6481 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-ca-configmap\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:47.941029 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.941001    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "kube-api-access-t7rsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:43:47.941143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.941108    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:43:47.941211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.941159    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:43:47.941407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.941385    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:43:47.941816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.941791    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:43:47.941916 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:47.941899    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig" pod "a25d4e45-29be-4581-b786-68e95371cde6" (UID: "a25d4e45-29be-4581-b786-68e95371cde6"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:43:48.040183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.040153    6481 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25d4e45-29be-4581-b786-68e95371cde6-local-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:48.040183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.040181    6481 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-csr-dns-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:48.040183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.040191    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:48.040635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.040199    6481 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-cacerts\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:48.040635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.040207    6481 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a25d4e45-29be-4581-b786-68e95371cde6-istio-kubeconfig\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:48.040635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.040215    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7rsm\" (UniqueName: \"kubernetes.io/projected/a25d4e45-29be-4581-b786-68e95371cde6-kube-api-access-t7rsm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:48.746223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.746190    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"
Nov 03 14:43:48.746223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.746215    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j" event={"ID":"a25d4e45-29be-4581-b786-68e95371cde6","Type":"ContainerDied","Data":"a6b640244fe1de327a523bb5d38927a1594dea41eccc867edfe5abd4eed22154"}
Nov 03 14:43:48.746462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.746284    6481 scope.go:122] "RemoveContainer" containerID="da721857c7598744f60fef2bcfb50aede42d1210a808c374a54b019d17800cd5"
Nov 03 14:43:48.810553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.810523    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"]
Nov 03 14:43:48.822513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:48.822488    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r9t9j"]
Nov 03 14:43:49.011103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:49.010993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:50.066627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:50.066595    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25d4e45-29be-4581-b786-68e95371cde6" path="/var/lib/kubelet/pods/a25d4e45-29be-4581-b786-68e95371cde6/volumes"
Nov 03 14:43:51.009090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:51.009032    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:53.109066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.109016    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:53.150215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.150183    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"]
Nov 03 14:43:53.150647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.150634    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a25d4e45-29be-4581-b786-68e95371cde6" containerName="discovery"
Nov 03 14:43:53.150693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.150648    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25d4e45-29be-4581-b786-68e95371cde6" containerName="discovery"
Nov 03 14:43:53.150754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.150745    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25d4e45-29be-4581-b786-68e95371cde6" containerName="discovery"
Nov 03 14:43:53.155264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.155223    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.174563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.174535    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"]
Nov 03 14:43:53.182949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.182923    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:43:53.183088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.182966    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:43:53.183088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.182966    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:43:53.183088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.183025    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:43:53.295995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.295954    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.295995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.295998    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lccx9\" (UniqueName: \"kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.296213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.296133    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.396707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.396632    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.396852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.396735    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.396852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.396756    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lccx9\" (UniqueName: \"kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.397380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.397354    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.398966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.398944    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.418354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.418327    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lccx9\" (UniqueName: \"kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9\") pod \"maas-api-revoke-sub-4cf995c2-nprpq\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.467715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.467691    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:53.771265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:53.771243    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"]
Nov 03 14:43:53.773860 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:43:53.773829    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a35533f_d107_42ec_b252_df501d6e69e5.slice/crio-4853ee621c50e5e86d007632ed21f26a20a4bdf114a72e73b8bfeff39be9337c WatchSource:0}: Error finding container 4853ee621c50e5e86d007632ed21f26a20a4bdf114a72e73b8bfeff39be9337c: Status 404 returned error can't find the container with id 4853ee621c50e5e86d007632ed21f26a20a4bdf114a72e73b8bfeff39be9337c
Nov 03 14:43:54.782499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:54.782463    6481 generic.go:356] "Generic (PLEG): container finished" podID="2a35533f-d107-42ec-b252-df501d6e69e5" containerID="6d0dde4df26e708c8e7604a9be0090db20de41336e17e761f98e21829430aecd" exitCode=0
Nov 03 14:43:54.782921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:54.782549    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq" event={"ID":"2a35533f-d107-42ec-b252-df501d6e69e5","Type":"ContainerDied","Data":"6d0dde4df26e708c8e7604a9be0090db20de41336e17e761f98e21829430aecd"}
Nov 03 14:43:54.782921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:54.782601    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq" event={"ID":"2a35533f-d107-42ec-b252-df501d6e69e5","Type":"ContainerStarted","Data":"4853ee621c50e5e86d007632ed21f26a20a4bdf114a72e73b8bfeff39be9337c"}
Nov 03 14:43:55.095347 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:55.095318    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:55.920142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:55.920119    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:56.024865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.024836    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca\") pod \"2a35533f-d107-42ec-b252-df501d6e69e5\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") "
Nov 03 14:43:56.025042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.024962    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9\" (UniqueName: \"kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9\") pod \"2a35533f-d107-42ec-b252-df501d6e69e5\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") "
Nov 03 14:43:56.025042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.024986    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca\") pod \"2a35533f-d107-42ec-b252-df501d6e69e5\" (UID: \"2a35533f-d107-42ec-b252-df501d6e69e5\") "
Nov 03 14:43:56.027553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.025972    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca" pod "2a35533f-d107-42ec-b252-df501d6e69e5" (UID: "2a35533f-d107-42ec-b252-df501d6e69e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:43:56.029273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.027898    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca" pod "2a35533f-d107-42ec-b252-df501d6e69e5" (UID: "2a35533f-d107-42ec-b252-df501d6e69e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:43:56.029273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.028165    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9" pod "2a35533f-d107-42ec-b252-df501d6e69e5" (UID: "2a35533f-d107-42ec-b252-df501d6e69e5"). InnerVolumeSpecName "kube-api-access-lccx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:43:56.127155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.127130    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:56.127155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.127154    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lccx9\" (UniqueName: \"kubernetes.io/projected/2a35533f-d107-42ec-b252-df501d6e69e5-kube-api-access-lccx9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:56.127322 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.127164    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a35533f-d107-42ec-b252-df501d6e69e5-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:43:56.795732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.795701    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"
Nov 03 14:43:56.795946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.795735    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq" event={"ID":"2a35533f-d107-42ec-b252-df501d6e69e5","Type":"ContainerDied","Data":"4853ee621c50e5e86d007632ed21f26a20a4bdf114a72e73b8bfeff39be9337c"}
Nov 03 14:43:56.795946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:56.795786    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4853ee621c50e5e86d007632ed21f26a20a4bdf114a72e73b8bfeff39be9337c"
Nov 03 14:43:57.110843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:57.110750    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:43:57.985534 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:43:57.985494    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:43:59.095739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:43:59.095697    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:00.161340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:00.161310    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:00.389644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:00.389604    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:01.016139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:01.016104    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:01.240884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:01.240844    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:03.092041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.091994    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:03.894146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.890813    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"]
Nov 03 14:44:03.894146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.891797    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2a35533f-d107-42ec-b252-df501d6e69e5" containerName="revoke-keys"
Nov 03 14:44:03.894146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.891818    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a35533f-d107-42ec-b252-df501d6e69e5" containerName="revoke-keys"
Nov 03 14:44:03.894146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.892115    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a35533f-d107-42ec-b252-df501d6e69e5" containerName="revoke-keys"
Nov 03 14:44:03.897048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.897023    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:03.912476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.912453    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"]
Nov 03 14:44:03.921865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.921837    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:44:03.922040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.922005    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:44:03.923937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.923916    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:44:03.924042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:03.923972    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:44:04.002581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.002552    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qzt2\" (UniqueName: \"kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.002709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.002589    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.002709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.002617    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.103496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.103468    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.103848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.103631    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qzt2\" (UniqueName: \"kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.103848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.103669    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.104264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.104218    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.105995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.105974    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.124340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.124314    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qzt2\" (UniqueName: \"kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2\") pod \"maas-api-revoke-sub-9140577d-km2wp\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.209900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.209866    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:04.512018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.511995    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"]
Nov 03 14:44:04.513594 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:04.513567    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3e51ca_e926_42ba_bb51_4c5e129bc314.slice/crio-50b0b57ea136a6a78ce8ea04c1f2158690192361318f4991c6f4800bcab62197 WatchSource:0}: Error finding container 50b0b57ea136a6a78ce8ea04c1f2158690192361318f4991c6f4800bcab62197: Status 404 returned error can't find the container with id 50b0b57ea136a6a78ce8ea04c1f2158690192361318f4991c6f4800bcab62197
Nov 03 14:44:04.842602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.842570    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp" event={"ID":"bb3e51ca-e926-42ba-bb51-4c5e129bc314","Type":"ContainerStarted","Data":"b2189d98e938007dd4001480331b3017ab7fb8bca2ca80e2aae40df51926a262"}
Nov 03 14:44:04.842744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.842629    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp" event={"ID":"bb3e51ca-e926-42ba-bb51-4c5e129bc314","Type":"ContainerStarted","Data":"50b0b57ea136a6a78ce8ea04c1f2158690192361318f4991c6f4800bcab62197"}
Nov 03 14:44:04.883962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:04.883915    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp" podStartSLOduration=1.883900532 podStartE2EDuration="1.883900532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:04.882860627 +0000 UTC m=+1922.932891817" watchObservedRunningTime="2026-11-03 14:44:04.883900532 +0000 UTC m=+1922.933931746"
Nov 03 14:44:05.053663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:05.053630    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:05.108815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:05.108735    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:05.849243 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:05.849202    6481 generic.go:356] "Generic (PLEG): container finished" podID="bb3e51ca-e926-42ba-bb51-4c5e129bc314" containerID="b2189d98e938007dd4001480331b3017ab7fb8bca2ca80e2aae40df51926a262" exitCode=0
Nov 03 14:44:05.849426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:05.849286    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp" event={"ID":"bb3e51ca-e926-42ba-bb51-4c5e129bc314","Type":"ContainerDied","Data":"b2189d98e938007dd4001480331b3017ab7fb8bca2ca80e2aae40df51926a262"}
Nov 03 14:44:06.994626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:06.994599    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:07.120256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.120127    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:07.135290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.135264    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca\") pod \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") "
Nov 03 14:44:07.135425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.135293    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2\" (UniqueName: \"kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2\") pod \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") "
Nov 03 14:44:07.135469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.135420    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca\") pod \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\" (UID: \"bb3e51ca-e926-42ba-bb51-4c5e129bc314\") "
Nov 03 14:44:07.135705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.135683    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca" pod "bb3e51ca-e926-42ba-bb51-4c5e129bc314" (UID: "bb3e51ca-e926-42ba-bb51-4c5e129bc314"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:07.137440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.137421    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2" pod "bb3e51ca-e926-42ba-bb51-4c5e129bc314" (UID: "bb3e51ca-e926-42ba-bb51-4c5e129bc314"). InnerVolumeSpecName "kube-api-access-5qzt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:07.137560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.137546    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca" pod "bb3e51ca-e926-42ba-bb51-4c5e129bc314" (UID: "bb3e51ca-e926-42ba-bb51-4c5e129bc314"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:44:07.236489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.236462    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:07.236489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.236487    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb3e51ca-e926-42ba-bb51-4c5e129bc314-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:07.236693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.236498    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qzt2\" (UniqueName: \"kubernetes.io/projected/bb3e51ca-e926-42ba-bb51-4c5e129bc314-kube-api-access-5qzt2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:07.742497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.742466    6481 scope.go:122] "RemoveContainer" containerID="56db2c3ce9623faabc2fe83b1bc7e227c8205e15e1e1aa13a1320fe93d21087c"
Nov 03 14:44:07.754087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.754063    6481 scope.go:122] "RemoveContainer" containerID="9e7092408b14a7749879f9e05bbdcc9a7f11659e4909a5f98000c4f9eb2c9fa9"
Nov 03 14:44:07.762392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.762374    6481 scope.go:122] "RemoveContainer" containerID="0683471168b0ee1cc26d56aec2157b733ecc35c14f59cbec416c32cbb0150164"
Nov 03 14:44:07.770027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.770008    6481 scope.go:122] "RemoveContainer" containerID="c75614a49fd6b5a0e448d6e7c876eedb3c58b071d9ac5f7d60cd22075145338b"
Nov 03 14:44:07.778129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.778105    6481 scope.go:122] "RemoveContainer" containerID="e572192d405d55b9af8fb938a78def0412e1d28d9dfbb200bc053045dfa8b06c"
Nov 03 14:44:07.860670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.860643    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"
Nov 03 14:44:07.860821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.860678    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp" event={"ID":"bb3e51ca-e926-42ba-bb51-4c5e129bc314","Type":"ContainerDied","Data":"50b0b57ea136a6a78ce8ea04c1f2158690192361318f4991c6f4800bcab62197"}
Nov 03 14:44:07.860821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:07.860722    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b0b57ea136a6a78ce8ea04c1f2158690192361318f4991c6f4800bcab62197"
Nov 03 14:44:09.092644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:09.092613    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:11.049219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:11.049172    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:12.922176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:12.922141    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:14.918047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:14.918010    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:15.795564 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.794669    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:15.825447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.825418    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:15.994005 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.993965    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"]
Nov 03 14:44:15.994657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.994636    6481 cpu_manager.go:414] "containerMap: removing container" podUID="bb3e51ca-e926-42ba-bb51-4c5e129bc314" containerName="revoke-keys"
Nov 03 14:44:15.994752 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.994659    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3e51ca-e926-42ba-bb51-4c5e129bc314" containerName="revoke-keys"
Nov 03 14:44:15.994811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.994781    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3e51ca-e926-42ba-bb51-4c5e129bc314" containerName="revoke-keys"
Nov 03 14:44:15.999644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:15.999624    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.016900 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.016874    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.017066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.017047    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjrw\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.017127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.017093    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.017195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.017180    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.020425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.020399    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"]
Nov 03 14:44:16.086898 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.086795    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:16.118045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.118002    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.118276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.118060    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.118276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.118170    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjrw\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.118417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.118285    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.118880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.118855    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.118998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.118895    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.120866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.120843    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.138996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.138971    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjrw\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.312710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.312685    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.593780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.593752    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:16.614729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.614704    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"]
Nov 03 14:44:16.616487 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:16.616461    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193f7d6b_30ed_42dc_9513_303b10755a04.slice/crio-efff9755218a02522940cb7e568f24dc9c6dd4479ebd97ce9c63a5c6798b5761 WatchSource:0}: Error finding container efff9755218a02522940cb7e568f24dc9c6dd4479ebd97ce9c63a5c6798b5761: Status 404 returned error can't find the container with id efff9755218a02522940cb7e568f24dc9c6dd4479ebd97ce9c63a5c6798b5761
Nov 03 14:44:16.915128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.915094    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52" event={"ID":"193f7d6b-30ed-42dc-9513-303b10755a04","Type":"ContainerStarted","Data":"1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64"}
Nov 03 14:44:16.915318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.915146    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52" event={"ID":"193f7d6b-30ed-42dc-9513-303b10755a04","Type":"ContainerStarted","Data":"efff9755218a02522940cb7e568f24dc9c6dd4479ebd97ce9c63a5c6798b5761"}
Nov 03 14:44:16.915318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.915168    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:16.920359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.920339    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:16.957820 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.957778    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52" podStartSLOduration=1.9577665629999998 podStartE2EDuration="1.957766563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:16.956800147 +0000 UTC m=+1935.006831345" watchObservedRunningTime="2026-11-03 14:44:16.957766563 +0000 UTC m=+1935.007797786"
Nov 03 14:44:16.986139 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.986114    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"]
Nov 03 14:44:16.989843 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:16.989829    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.010356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.010319    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-3e061afe7424\""
Nov 03 14:44:17.010356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.010326    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-3f3c5a621d3c\""
Nov 03 14:44:17.010356 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.010347    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"agc-cred-78c77d81649c\""
Nov 03 14:44:17.010806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.010430    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-e2e--29c79c46-dockercfg-bqq2f\""
Nov 03 14:44:17.012358 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.012339    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"]
Nov 03 14:44:17.013017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.012996    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b4361f\""
Nov 03 14:44:17.013017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.013010    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"routing-overlay\""
Nov 03 14:44:17.013170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.013033    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"kube-root-ca.crt\""
Nov 03 14:44:17.013170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.013087    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b4361f\"/\"openshift-service-ca.crt\""
Nov 03 14:44:17.028607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.028577    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgf9\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.028798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.028630    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.028798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.028659    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.028798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.028780    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.130487 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.130450    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgf9\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.130727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.130513    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.130727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.130545    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.130727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.130584    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.131725 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.131696    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.131895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.131761    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.134007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.133982    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.151424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.151403    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgf9\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.302417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.302345    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.604104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.604073    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"]
Nov 03 14:44:17.605499 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:17.605474    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5e5810_b741_4540_ae45_5960c9167fa4.slice/crio-69a795b34e64e244fcd49c283fc352dedd68dba6aa63fddf6d7efa5ed688390e WatchSource:0}: Error finding container 69a795b34e64e244fcd49c283fc352dedd68dba6aa63fddf6d7efa5ed688390e: Status 404 returned error can't find the container with id 69a795b34e64e244fcd49c283fc352dedd68dba6aa63fddf6d7efa5ed688390e
Nov 03 14:44:17.922736 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.922697    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" event={"ID":"9a5e5810-b741-4540-ae45-5960c9167fa4","Type":"ContainerStarted","Data":"0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624"}
Nov 03 14:44:17.922942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.922768    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" event={"ID":"9a5e5810-b741-4540-ae45-5960c9167fa4","Type":"ContainerStarted","Data":"69a795b34e64e244fcd49c283fc352dedd68dba6aa63fddf6d7efa5ed688390e"}
Nov 03 14:44:17.922942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.922889    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:17.965972 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:17.965930    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" podStartSLOduration=1.965916253 podStartE2EDuration="1.965916253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:17.964603794 +0000 UTC m=+1936.014634977" watchObservedRunningTime="2026-11-03 14:44:17.965916253 +0000 UTC m=+1936.015947471"
Nov 03 14:44:18.873611 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:18.873578    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:19.049448 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:19.049405    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-3f3c5a621d3c: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:19.049448 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:19.049436    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:19.049695 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:19.049500    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials podName:9a5e5810-b741-4540-ae45-5960c9167fa4 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:19.549484272 +0000 UTC m=+1937.599515446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4") : secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:19.100973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.100940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:19.389283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.389255    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"]
Nov 03 14:44:19.393263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.393246    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.412045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.412021    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"]
Nov 03 14:44:19.421047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.421019    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:44:19.421276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.421260    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:44:19.421276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.421268    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:44:19.421440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.421320    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:44:19.454508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.454482    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.454650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.454533    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.454695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.454668    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bvh\" (UniqueName: \"kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.555685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.555654    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.555853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.555725    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bvh\" (UniqueName: \"kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.555853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.555805    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.555981 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:19.555904    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-3f3c5a621d3c: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:19.555981 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:19.555927    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:19.556086 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:19.555991    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials podName:9a5e5810-b741-4540-ae45-5960c9167fa4 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:20.555973206 +0000 UTC m=+1938.606004368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4") : secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:19.556538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.556519    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.557977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.557960    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.576793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.576760    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bvh\" (UniqueName: \"kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh\") pod \"maas-api-revoke-sub-459bfecc-fp7pv\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.652562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.652501    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"]
Nov 03 14:44:19.683282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.683253    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"]
Nov 03 14:44:19.687819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.687796    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.724506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.724475    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:19.741403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.741372    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"]
Nov 03 14:44:19.757569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.757536    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.757714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.757609    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.757714 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.757666    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.757832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.757811    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.866269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.863680    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.866269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.863751    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.866269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.863912    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.866269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.863979    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.866269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.864811    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.866269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.865853    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.874159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.874110    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:19.886772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:19.886726    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:20.036654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.036626    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:20.095086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.093369    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"]
Nov 03 14:44:20.345931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.345906    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"]
Nov 03 14:44:20.347084 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:20.347045    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1512f5d3_14c5_4d0f_ae08_4a443798d18f.slice/crio-a78e30e1d081a69cd8ca51be2123243bb8f454d1bc8989af9e39e0d0e9920f55 WatchSource:0}: Error finding container a78e30e1d081a69cd8ca51be2123243bb8f454d1bc8989af9e39e0d0e9920f55: Status 404 returned error can't find the container with id a78e30e1d081a69cd8ca51be2123243bb8f454d1bc8989af9e39e0d0e9920f55
Nov 03 14:44:20.550658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.550627    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"]
Nov 03 14:44:20.571980 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:20.571911    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-3f3c5a621d3c: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:20.571980 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:20.571940    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:20.572158 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:20.571995    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials podName:9a5e5810-b741-4540-ae45-5960c9167fa4 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:22.571977412 +0000 UTC m=+1940.622008589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4") : secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:20.583301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.583276    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"]
Nov 03 14:44:20.587412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.587396    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.608288 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.608265    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"]
Nov 03 14:44:20.672508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.672475    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.672680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.672533    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.672680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.672611    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbz9q\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.672680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.672655    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.773672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.773640    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.773822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.773695    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.773822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.773741    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbz9q\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.773822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.773793    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.774521 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.774501    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.774599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.774582    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.776125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.776091    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.794709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.794684    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbz9q\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.866669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.866600    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:20.901632 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.901610    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:20.946342 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.946308    6481 generic.go:356] "Generic (PLEG): container finished" podID="39953a3d-b932-4503-91e7-90f91d010a31" containerID="6f1f99911dc9ab1796c3cf85ccf0a3ce6e23ff7c5ddd5a75c38b500d7ce2d538" exitCode=0
Nov 03 14:44:20.946529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.946386    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv" event={"ID":"39953a3d-b932-4503-91e7-90f91d010a31","Type":"ContainerDied","Data":"6f1f99911dc9ab1796c3cf85ccf0a3ce6e23ff7c5ddd5a75c38b500d7ce2d538"}
Nov 03 14:44:20.946529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.946446    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv" event={"ID":"39953a3d-b932-4503-91e7-90f91d010a31","Type":"ContainerStarted","Data":"71a68151a2809b73b2f9ed3fba14078a9c18f4f5dff4fdbb04edf10fb64c8745"}
Nov 03 14:44:20.948185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.948161    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9" event={"ID":"1512f5d3-14c5-4d0f-ae08-4a443798d18f","Type":"ContainerStarted","Data":"50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5"}
Nov 03 14:44:20.948325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.948217    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:20.948325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:20.948268    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9" event={"ID":"1512f5d3-14c5-4d0f-ae08-4a443798d18f","Type":"ContainerStarted","Data":"a78e30e1d081a69cd8ca51be2123243bb8f454d1bc8989af9e39e0d0e9920f55"}
Nov 03 14:44:21.031323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.031273    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9" podStartSLOduration=2.031258761 podStartE2EDuration="2.031258761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:21.029086106 +0000 UTC m=+1939.079117290" watchObservedRunningTime="2026-11-03 14:44:21.031258761 +0000 UTC m=+1939.081289940"
Nov 03 14:44:21.084332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.084299    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:21.202992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.202964    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"]
Nov 03 14:44:21.203826 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:21.203798    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7c0fe9_d2a1_4cb5_a340_7316d9dcb065.slice/crio-6fe35b6cf25f753a4d900aee14d55501b7917ecd04d7bbe3879021ec53217a78 WatchSource:0}: Error finding container 6fe35b6cf25f753a4d900aee14d55501b7917ecd04d7bbe3879021ec53217a78: Status 404 returned error can't find the container with id 6fe35b6cf25f753a4d900aee14d55501b7917ecd04d7bbe3879021ec53217a78
Nov 03 14:44:21.308141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.308116    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:21.955789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.955757    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr" event={"ID":"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065","Type":"ContainerStarted","Data":"fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae"}
Nov 03 14:44:21.956268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.955811    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr" event={"ID":"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065","Type":"ContainerStarted","Data":"6fe35b6cf25f753a4d900aee14d55501b7917ecd04d7bbe3879021ec53217a78"}
Nov 03 14:44:21.956268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:21.956177    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:22.001774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.001710    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr" podStartSLOduration=2.001689604 podStartE2EDuration="2.001689604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:21.998423424 +0000 UTC m=+1940.048454611" watchObservedRunningTime="2026-11-03 14:44:22.001689604 +0000 UTC m=+1940.051720789"
Nov 03 14:44:22.095527 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.095503    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:22.190238 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.190197    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca\") pod \"39953a3d-b932-4503-91e7-90f91d010a31\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") "
Nov 03 14:44:22.190402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.190326    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca\") pod \"39953a3d-b932-4503-91e7-90f91d010a31\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") "
Nov 03 14:44:22.190402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.190365    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh\" (UniqueName: \"kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh\") pod \"39953a3d-b932-4503-91e7-90f91d010a31\" (UID: \"39953a3d-b932-4503-91e7-90f91d010a31\") "
Nov 03 14:44:22.190610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.190589    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca" pod "39953a3d-b932-4503-91e7-90f91d010a31" (UID: "39953a3d-b932-4503-91e7-90f91d010a31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:22.190844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.190820    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:22.192440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.192416    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca" pod "39953a3d-b932-4503-91e7-90f91d010a31" (UID: "39953a3d-b932-4503-91e7-90f91d010a31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:44:22.192534 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.192477    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh" pod "39953a3d-b932-4503-91e7-90f91d010a31" (UID: "39953a3d-b932-4503-91e7-90f91d010a31"). InnerVolumeSpecName "kube-api-access-x9bvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:22.291673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.291615    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39953a3d-b932-4503-91e7-90f91d010a31-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:22.291673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.291638    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bvh\" (UniqueName: \"kubernetes.io/projected/39953a3d-b932-4503-91e7-90f91d010a31-kube-api-access-x9bvh\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:22.594664 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:22.594592    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-b4361f/agc-cred-3f3c5a621d3c: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:22.594664 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:22.594619    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6: secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:22.594835 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:22.594670    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials podName:9a5e5810-b741-4540-ae45-5960c9167fa4 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:26.594658061 +0000 UTC m=+1944.644689221 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4") : secret "agc-cred-3f3c5a621d3c" not found
Nov 03 14:44:22.751569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.751539    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:22.962453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.962421    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"
Nov 03 14:44:22.962892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.962449    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv" event={"ID":"39953a3d-b932-4503-91e7-90f91d010a31","Type":"ContainerDied","Data":"71a68151a2809b73b2f9ed3fba14078a9c18f4f5dff4fdbb04edf10fb64c8745"}
Nov 03 14:44:22.962892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:22.962496    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71a68151a2809b73b2f9ed3fba14078a9c18f4f5dff4fdbb04edf10fb64c8745"
Nov 03 14:44:23.105835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:23.105799    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:24.653311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.653275    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52" podUID="193f7d6b-30ed-42dc-9513-303b10755a04" containerName="payload-processing" containerID="cri-o://1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64" gracePeriod=25
Nov 03 14:44:24.910361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.910338    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:24.975647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.975618    6481 generic.go:356] "Generic (PLEG): container finished" podID="193f7d6b-30ed-42dc-9513-303b10755a04" containerID="1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64" exitCode=0
Nov 03 14:44:24.975797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.975654    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"
Nov 03 14:44:24.975797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.975694    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52" event={"ID":"193f7d6b-30ed-42dc-9513-303b10755a04","Type":"ContainerDied","Data":"1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64"}
Nov 03 14:44:24.975797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.975748    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52" event={"ID":"193f7d6b-30ed-42dc-9513-303b10755a04","Type":"ContainerDied","Data":"efff9755218a02522940cb7e568f24dc9c6dd4479ebd97ce9c63a5c6798b5761"}
Nov 03 14:44:24.975797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.975774    6481 scope.go:122] "RemoveContainer" containerID="1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64"
Nov 03 14:44:24.984816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.984796    6481 scope.go:122] "RemoveContainer" containerID="1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64"
Nov 03 14:44:24.985102 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:24.985077    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64\": container with ID starting with 1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64 not found: ID does not exist" containerID="1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64"
Nov 03 14:44:24.985170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:24.985106    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64"} err="failed to get container status \"1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64\": rpc error: code = NotFound desc = could not find container \"1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64\": container with ID starting with 1a25de00b396edebb6f8667a8ee76b907f6ebb4e4017326553c1025d34ed9b64 not found: ID does not exist"
Nov 03 14:44:25.019706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.019683    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume\") pod \"193f7d6b-30ed-42dc-9513-303b10755a04\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") "
Nov 03 14:44:25.019853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.019834    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials\") pod \"193f7d6b-30ed-42dc-9513-303b10755a04\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") "
Nov 03 14:44:25.019930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.019879    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw\") pod \"193f7d6b-30ed-42dc-9513-303b10755a04\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") "
Nov 03 14:44:25.019991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.019953    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume\") pod \"193f7d6b-30ed-42dc-9513-303b10755a04\" (UID: \"193f7d6b-30ed-42dc-9513-303b10755a04\") "
Nov 03 14:44:25.020114 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.020093    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume" pod "193f7d6b-30ed-42dc-9513-303b10755a04" (UID: "193f7d6b-30ed-42dc-9513-303b10755a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:25.020362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.020336    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume" pod "193f7d6b-30ed-42dc-9513-303b10755a04" (UID: "193f7d6b-30ed-42dc-9513-303b10755a04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:25.020431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.020392    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:25.022078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.022056    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials" pod "193f7d6b-30ed-42dc-9513-303b10755a04" (UID: "193f7d6b-30ed-42dc-9513-303b10755a04"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:25.022172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.022135    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw" pod "193f7d6b-30ed-42dc-9513-303b10755a04" (UID: "193f7d6b-30ed-42dc-9513-303b10755a04"). InnerVolumeSpecName "kube-api-access-psjrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:25.051091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.051065    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"]
Nov 03 14:44:25.054928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.054908    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-703f6a22-gm9js"]
Nov 03 14:44:25.105465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.105438    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:25.122010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.121986    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:25.122010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.122009    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psjrw\" (UniqueName: \"kubernetes.io/projected/193f7d6b-30ed-42dc-9513-303b10755a04-kube-api-access-psjrw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:25.122174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.122022    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193f7d6b-30ed-42dc-9513-303b10755a04-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:25.322240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.322202    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"]
Nov 03 14:44:25.333935 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.333907    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-99d4w9j52"]
Nov 03 14:44:25.551977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.551896    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" podUID="9a5e5810-b741-4540-ae45-5960c9167fa4" containerName="payload-processing" containerID="cri-o://0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624" gracePeriod=25
Nov 03 14:44:25.810785 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.810721    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:25.930279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930251    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials\") pod \"9a5e5810-b741-4540-ae45-5960c9167fa4\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") "
Nov 03 14:44:25.930418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930304    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume\") pod \"9a5e5810-b741-4540-ae45-5960c9167fa4\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") "
Nov 03 14:44:25.930418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930330    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9\") pod \"9a5e5810-b741-4540-ae45-5960c9167fa4\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") "
Nov 03 14:44:25.930418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930388    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume\") pod \"9a5e5810-b741-4540-ae45-5960c9167fa4\" (UID: \"9a5e5810-b741-4540-ae45-5960c9167fa4\") "
Nov 03 14:44:25.930814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930782    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume" pod "9a5e5810-b741-4540-ae45-5960c9167fa4" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:25.930960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930878    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:25.930960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.930889    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume" pod "9a5e5810-b741-4540-ae45-5960c9167fa4" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:25.932330 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.932310    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials" pod "9a5e5810-b741-4540-ae45-5960c9167fa4" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:25.932461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.932445    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9" pod "9a5e5810-b741-4540-ae45-5960c9167fa4" (UID: "9a5e5810-b741-4540-ae45-5960c9167fa4"). InnerVolumeSpecName "kube-api-access-wzgf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:25.983887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.983858    6481 generic.go:356] "Generic (PLEG): container finished" podID="9a5e5810-b741-4540-ae45-5960c9167fa4" containerID="0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624" exitCode=0
Nov 03 14:44:25.984017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.983902    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"
Nov 03 14:44:25.984017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.983945    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" event={"ID":"9a5e5810-b741-4540-ae45-5960c9167fa4","Type":"ContainerDied","Data":"0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624"}
Nov 03 14:44:25.984085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.984024    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6" event={"ID":"9a5e5810-b741-4540-ae45-5960c9167fa4","Type":"ContainerDied","Data":"69a795b34e64e244fcd49c283fc352dedd68dba6aa63fddf6d7efa5ed688390e"}
Nov 03 14:44:25.984085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.984050    6481 scope.go:122] "RemoveContainer" containerID="0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624"
Nov 03 14:44:25.993257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.993223    6481 scope.go:122] "RemoveContainer" containerID="0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624"
Nov 03 14:44:25.993539 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:25.993524    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624\": container with ID starting with 0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624 not found: ID does not exist" containerID="0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624"
Nov 03 14:44:25.993592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:25.993546    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624"} err="failed to get container status \"0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624\": rpc error: code = NotFound desc = could not find container \"0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624\": container with ID starting with 0a487386c9aa4e4899c69fa658b90a1702621b2ea5825b8ad2dfdc6f8c4e3624 not found: ID does not exist"
Nov 03 14:44:26.031638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.031609    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9a5e5810-b741-4540-ae45-5960c9167fa4-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:26.031638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.031633    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:26.031774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.031644    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgf9\" (UniqueName: \"kubernetes.io/projected/9a5e5810-b741-4540-ae45-5960c9167fa4-kube-api-access-wzgf9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:26.037859 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.037838    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"]
Nov 03 14:44:26.045372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.045350    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-8fd8kw9b6"]
Nov 03 14:44:26.066296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.066219    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc" path="/var/lib/kubelet/pods/0e93ed3a-bd20-4a2c-9be6-d41ce1f46ffc/volumes"
Nov 03 14:44:26.066643 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.066629    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193f7d6b-30ed-42dc-9513-303b10755a04" path="/var/lib/kubelet/pods/193f7d6b-30ed-42dc-9513-303b10755a04/volumes"
Nov 03 14:44:26.066971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:26.066960    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5e5810-b741-4540-ae45-5960c9167fa4" path="/var/lib/kubelet/pods/9a5e5810-b741-4540-ae45-5960c9167fa4/volumes"
Nov 03 14:44:27.111164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:27.111120    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:27.985438 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:27.985406    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:44:28.155837 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.155571    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-084a61fd1674: secret "agc-cred-084a61fd1674" not found
Nov 03 14:44:28.155837 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.155609    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b0be02ea2a5b: secret "agc-cred-b0be02ea2a5b" not found
Nov 03 14:44:28.155837 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.155622    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm: [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:28.155837 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.155694    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials podName:9ab6ed9b-4aeb-4e61-b15f-c27b642fc243 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:28.655673734 +0000 UTC m=+1946.705704909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243") : [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:28.661542 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.661511    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-084a61fd1674: secret "agc-cred-084a61fd1674" not found
Nov 03 14:44:28.661542 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.661545    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b0be02ea2a5b: secret "agc-cred-b0be02ea2a5b" not found
Nov 03 14:44:28.661768 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.661554    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm: [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:28.661768 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:28.661615    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials podName:9ab6ed9b-4aeb-4e61-b15f-c27b642fc243 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:29.661600691 +0000 UTC m=+1947.711631851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243") : [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:29.111469 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:29.111393    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:29.672809 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:29.672779    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-084a61fd1674: secret "agc-cred-084a61fd1674" not found
Nov 03 14:44:29.672809 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:29.672811    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b0be02ea2a5b: secret "agc-cred-b0be02ea2a5b" not found
Nov 03 14:44:29.673249 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:29.672820    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm: [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:29.673249 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:29.672874    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials podName:9ab6ed9b-4aeb-4e61-b15f-c27b642fc243 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:31.672857248 +0000 UTC m=+1949.722888413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243") : [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:30.242424 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242397    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"]
Nov 03 14:44:30.242855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242843    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a5e5810-b741-4540-ae45-5960c9167fa4" containerName="payload-processing"
Nov 03 14:44:30.242903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242857    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5e5810-b741-4540-ae45-5960c9167fa4" containerName="payload-processing"
Nov 03 14:44:30.242903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242876    6481 cpu_manager.go:414] "containerMap: removing container" podUID="39953a3d-b932-4503-91e7-90f91d010a31" containerName="revoke-keys"
Nov 03 14:44:30.242903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242882    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39953a3d-b932-4503-91e7-90f91d010a31" containerName="revoke-keys"
Nov 03 14:44:30.242903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242901    6481 cpu_manager.go:414] "containerMap: removing container" podUID="193f7d6b-30ed-42dc-9513-303b10755a04" containerName="payload-processing"
Nov 03 14:44:30.243034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242906    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193f7d6b-30ed-42dc-9513-303b10755a04" containerName="payload-processing"
Nov 03 14:44:30.243034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242967    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5e5810-b741-4540-ae45-5960c9167fa4" containerName="payload-processing"
Nov 03 14:44:30.243034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242977    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="39953a3d-b932-4503-91e7-90f91d010a31" containerName="revoke-keys"
Nov 03 14:44:30.243034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.242985    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="193f7d6b-30ed-42dc-9513-303b10755a04" containerName="payload-processing"
Nov 03 14:44:30.247213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.247194    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.268396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.268374    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"]
Nov 03 14:44:30.276942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.276922    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:44:30.277068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.276980    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:44:30.277130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.277001    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:44:30.277130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.277019    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:44:30.379708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.379679    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.379893 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.379868    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.380016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.379997    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tggx\" (UniqueName: \"kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.481472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.481440    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.481645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.481524    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tggx\" (UniqueName: \"kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.481645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.481562    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.482093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.482069    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.483975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.483959    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.502294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.502212    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tggx\" (UniqueName: \"kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx\") pod \"maas-api-revoke-sub-f7399150-r4qvh\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.560315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.560287    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:30.863266 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:30.863221    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"]
Nov 03 14:44:30.864352 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:30.864327    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63233c16_25af_4833_bbd1_8f22a37075b1.slice/crio-ea58074ee73360f3e225c8eb9f1d2abfc717b36cb1ce002d85841af53071188b WatchSource:0}: Error finding container ea58074ee73360f3e225c8eb9f1d2abfc717b36cb1ce002d85841af53071188b: Status 404 returned error can't find the container with id ea58074ee73360f3e225c8eb9f1d2abfc717b36cb1ce002d85841af53071188b
Nov 03 14:44:31.018583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:31.018541    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh" event={"ID":"63233c16-25af-4833-bbd1-8f22a37075b1","Type":"ContainerStarted","Data":"ea58074ee73360f3e225c8eb9f1d2abfc717b36cb1ce002d85841af53071188b"}
Nov 03 14:44:31.102054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:31.102017    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:31.158529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:31.158498    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:31.695944 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:31.695912    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-084a61fd1674: secret "agc-cred-084a61fd1674" not found
Nov 03 14:44:31.695944 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:31.695944    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b0be02ea2a5b: secret "agc-cred-b0be02ea2a5b" not found
Nov 03 14:44:31.696156 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:31.695954    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm: [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:31.696156 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:31.696008    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials podName:9ab6ed9b-4aeb-4e61-b15f-c27b642fc243 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:35.695994559 +0000 UTC m=+1953.746025734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243") : [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:31.958804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:31.958734    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:32.026819 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:32.026789    6481 generic.go:356] "Generic (PLEG): container finished" podID="63233c16-25af-4833-bbd1-8f22a37075b1" containerID="b716d52ce9e0bed7349f588e534408ab0e43e34d6a9d5e91bc9d020439680f54" exitCode=0
Nov 03 14:44:32.027006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:32.026861    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh" event={"ID":"63233c16-25af-4833-bbd1-8f22a37075b1","Type":"ContainerDied","Data":"b716d52ce9e0bed7349f588e534408ab0e43e34d6a9d5e91bc9d020439680f54"}
Nov 03 14:44:32.036391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:32.036368    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"]
Nov 03 14:44:32.966019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:32.965988    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:44:33.109635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.109603    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:33.178801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.178770    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:33.313319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.313223    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca\") pod \"63233c16-25af-4833-bbd1-8f22a37075b1\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") "
Nov 03 14:44:33.313319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.313285    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca\") pod \"63233c16-25af-4833-bbd1-8f22a37075b1\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") "
Nov 03 14:44:33.313319 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.313320    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx\" (UniqueName: \"kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx\") pod \"63233c16-25af-4833-bbd1-8f22a37075b1\" (UID: \"63233c16-25af-4833-bbd1-8f22a37075b1\") "
Nov 03 14:44:33.313557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.313540    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca" pod "63233c16-25af-4833-bbd1-8f22a37075b1" (UID: "63233c16-25af-4833-bbd1-8f22a37075b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:33.313710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.313693    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:33.315480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.315455    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca" pod "63233c16-25af-4833-bbd1-8f22a37075b1" (UID: "63233c16-25af-4833-bbd1-8f22a37075b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:44:33.315598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.315518    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx" pod "63233c16-25af-4833-bbd1-8f22a37075b1" (UID: "63233c16-25af-4833-bbd1-8f22a37075b1"). InnerVolumeSpecName "kube-api-access-5tggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:33.414932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.414907    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63233c16-25af-4833-bbd1-8f22a37075b1-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:33.414932 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:33.414929    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tggx\" (UniqueName: \"kubernetes.io/projected/63233c16-25af-4833-bbd1-8f22a37075b1-kube-api-access-5tggx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:34.041251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:34.041199    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"
Nov 03 14:44:34.041746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:34.041199    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh" event={"ID":"63233c16-25af-4833-bbd1-8f22a37075b1","Type":"ContainerDied","Data":"ea58074ee73360f3e225c8eb9f1d2abfc717b36cb1ce002d85841af53071188b"}
Nov 03 14:44:34.041746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:34.041324    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea58074ee73360f3e225c8eb9f1d2abfc717b36cb1ce002d85841af53071188b"
Nov 03 14:44:35.035205 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:35.035173    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:35.741738 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:35.741706    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-084a61fd1674: secret "agc-cred-084a61fd1674" not found
Nov 03 14:44:35.741738 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:35.741741    6481 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-331c9e/agc-cred-b0be02ea2a5b: secret "agc-cred-b0be02ea2a5b" not found
Nov 03 14:44:35.742184 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:35.741750    6481 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm: [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:35.742184 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:35.741804    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials podName:9ab6ed9b-4aeb-4e61-b15f-c27b642fc243 nodeName:}" failed. No retries permitted until 2026-11-03 14:44:43.741790558 +0000 UTC m=+1961.791821719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243") : [secret "agc-cred-084a61fd1674" not found, secret "agc-cred-b0be02ea2a5b" not found]
Nov 03 14:44:36.994282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:36.994251    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:37.037344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.037305    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" podUID="9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" containerName="payload-processing" containerID="cri-o://de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e" gracePeriod=25
Nov 03 14:44:37.303560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.303523    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:44:37.459996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.459960    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh\") pod \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") "
Nov 03 14:44:37.460167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460048    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume\") pod \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") "
Nov 03 14:44:37.460261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460216    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume\") pod \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") "
Nov 03 14:44:37.460420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460400    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials\") pod \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\" (UID: \"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243\") "
Nov 03 14:44:37.460542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460409    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume" pod "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:37.460597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460510    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume" pod "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:37.460808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460787    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:37.460883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.460814    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:37.462187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.462166    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh" pod "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243"). InnerVolumeSpecName "kube-api-access-wz7wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:37.462274 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.462255    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials" pod "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" (UID: "9ab6ed9b-4aeb-4e61-b15f-c27b642fc243"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:37.562394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.562326    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:37.562394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:37.562350    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7wh\" (UniqueName: \"kubernetes.io/projected/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243-kube-api-access-wz7wh\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:38.063903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.063874    6481 generic.go:356] "Generic (PLEG): container finished" podID="9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" containerID="de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e" exitCode=0
Nov 03 14:44:38.064415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.063984    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"
Nov 03 14:44:38.066445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.066419    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" event={"ID":"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243","Type":"ContainerDied","Data":"de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e"}
Nov 03 14:44:38.066580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.066457    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm" event={"ID":"9ab6ed9b-4aeb-4e61-b15f-c27b642fc243","Type":"ContainerDied","Data":"f1e3b9dfaa57b7993e29c5df12dcba52ea3c5627fcfb36951d081002373febc1"}
Nov 03 14:44:38.066580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.066480    6481 scope.go:122] "RemoveContainer" containerID="de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e"
Nov 03 14:44:38.076057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.076037    6481 scope.go:122] "RemoveContainer" containerID="de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e"
Nov 03 14:44:38.076321 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:38.076303    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e\": container with ID starting with de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e not found: ID does not exist" containerID="de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e"
Nov 03 14:44:38.076382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.076328    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e"} err="failed to get container status \"de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e\": rpc error: code = NotFound desc = could not find container \"de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e\": container with ID starting with de0190fd725f92c507d68ab6a52fbde8c95ef918e7312b3ea906d8052893a00e not found: ID does not exist"
Nov 03 14:44:38.117261 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.117223    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"]
Nov 03 14:44:38.128729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:38.128707    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-946748crm"]
Nov 03 14:44:39.041560 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:39.041523    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"]
Nov 03 14:44:39.052059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:39.052033    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96fa1109-8qbwj"]
Nov 03 14:44:39.096547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:39.096493    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:40.066013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:40.065980    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f9621b6-465d-4b17-8f92-15d830277191" path="/var/lib/kubelet/pods/1f9621b6-465d-4b17-8f92-15d830277191/volumes"
Nov 03 14:44:40.066416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:40.066402    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" path="/var/lib/kubelet/pods/9ab6ed9b-4aeb-4e61-b15f-c27b642fc243/volumes"
Nov 03 14:44:41.109511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:41.109476    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:41.991305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:41.991272    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:42.036211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.036170    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.161992    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"]
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.163005    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" containerName="payload-processing"
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.163028    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" containerName="payload-processing"
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.163091    6481 cpu_manager.go:414] "containerMap: removing container" podUID="63233c16-25af-4833-bbd1-8f22a37075b1" containerName="revoke-keys"
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.163099    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63233c16-25af-4833-bbd1-8f22a37075b1" containerName="revoke-keys"
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.163354    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="63233c16-25af-4833-bbd1-8f22a37075b1" containerName="revoke-keys"
Nov 03 14:44:42.165508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.163388    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab6ed9b-4aeb-4e61-b15f-c27b642fc243" containerName="payload-processing"
Nov 03 14:44:42.169281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.169252    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.184420 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.184389    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"]
Nov 03 14:44:42.263494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.263193    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:42.311556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.311516    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmpq\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.311788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.311709    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.311788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.311768    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.311938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.311898    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.413398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.413349    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.413583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.413419    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.413583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.413544    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.413718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.413680    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmpq\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.414336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.414302    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.414494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.414424    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.416286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.416261    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.435071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.435039    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmpq\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq\") pod \"payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") " pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.483154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.483117    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:42.555314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.554861    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"]
Nov 03 14:44:42.559891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.559866    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.580076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.576176    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:44:42.580076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.577288    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:44:42.580076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.578784    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:44:42.580076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.578909    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:44:42.584715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.584692    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"]
Nov 03 14:44:42.716959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.716923    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.717121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.717096    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.717284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.717137    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hkc\" (UniqueName: \"kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.804022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.803992    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"]
Nov 03 14:44:42.805526 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:42.805465    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5517347_66e7_402c_b301_1e29c5dcbe8b.slice/crio-54788238878b56c8de972a17f8a2397cca56b5d13d4a485bfa05d26c31788b3f WatchSource:0}: Error finding container 54788238878b56c8de972a17f8a2397cca56b5d13d4a485bfa05d26c31788b3f: Status 404 returned error can't find the container with id 54788238878b56c8de972a17f8a2397cca56b5d13d4a485bfa05d26c31788b3f
Nov 03 14:44:42.818496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.818471    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hkc\" (UniqueName: \"kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.818605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.818516    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.818673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.818620    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.819258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.819218    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.820779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.820757    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.839077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.839058    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hkc\" (UniqueName: \"kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc\") pod \"maas-api-revoke-sub-5fecba05-gbv5f\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:42.889828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:42.889787    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:43.093069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.092988    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp" event={"ID":"e5517347-66e7-402c-b301-1e29c5dcbe8b","Type":"ContainerStarted","Data":"bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06"}
Nov 03 14:44:43.093069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.093042    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp" event={"ID":"e5517347-66e7-402c-b301-1e29c5dcbe8b","Type":"ContainerStarted","Data":"54788238878b56c8de972a17f8a2397cca56b5d13d4a485bfa05d26c31788b3f"}
Nov 03 14:44:43.093279 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.093099    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:43.130733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.130705    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"]
Nov 03 14:44:43.135649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.135630    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.136208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.136168    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp" podStartSLOduration=1.136155815 podStartE2EDuration="1.136155815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:43.134558276 +0000 UTC m=+1961.184589459" watchObservedRunningTime="2026-11-03 14:44:43.136155815 +0000 UTC m=+1961.186187025"
Nov 03 14:44:43.138675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.138653    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:43.154950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.154928    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"]
Nov 03 14:44:43.200163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.200140    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"]
Nov 03 14:44:43.201678 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:43.201654    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cfc825_e1b1_46e6_b6d0_65925d95b386.slice/crio-e2a7178d45984c1a972e374c01956e6eb27f3e44973178f717c441299088226b WatchSource:0}: Error finding container e2a7178d45984c1a972e374c01956e6eb27f3e44973178f717c441299088226b: Status 404 returned error can't find the container with id e2a7178d45984c1a972e374c01956e6eb27f3e44973178f717c441299088226b
Nov 03 14:44:43.323202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.323176    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpnjn\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.323311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.323214    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.323366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.323345    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.323423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.323396    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.424285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.424239    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpnjn\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.424389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.424308    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.424461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.424392    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.424461 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.424453    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.425268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.425243    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.425558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.425538    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.426676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.426658    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.445840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.445821    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpnjn\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn\") pod \"payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") " pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.448649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.448629    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:43.751553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:43.751517    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"]
Nov 03 14:44:43.752601 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:43.752577    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90933fd_b96a_4e6e_bc8e_f2accbc0df53.slice/crio-1184968523dfe4bd85d5b588879b5b64619a6f7a378d3500959100cd66f608b3 WatchSource:0}: Error finding container 1184968523dfe4bd85d5b588879b5b64619a6f7a378d3500959100cd66f608b3: Status 404 returned error can't find the container with id 1184968523dfe4bd85d5b588879b5b64619a6f7a378d3500959100cd66f608b3
Nov 03 14:44:44.100323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.100283    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd" event={"ID":"f90933fd-b96a-4e6e-bc8e-f2accbc0df53","Type":"ContainerStarted","Data":"503738e50797b66e0ccdcaa2c10a031af3481135347768b02b155d62bf3e75ed"}
Nov 03 14:44:44.100499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.100345    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd" event={"ID":"f90933fd-b96a-4e6e-bc8e-f2accbc0df53","Type":"ContainerStarted","Data":"1184968523dfe4bd85d5b588879b5b64619a6f7a378d3500959100cd66f608b3"}
Nov 03 14:44:44.100499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.100376    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:44.102056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.102033    6481 generic.go:356] "Generic (PLEG): container finished" podID="b6cfc825-e1b1-46e6-b6d0-65925d95b386" containerID="ae42cb848b6abb1114109e290a469a63b98879d1b154d0d9c9347879c35f3cc0" exitCode=0
Nov 03 14:44:44.102166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.102104    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f" event={"ID":"b6cfc825-e1b1-46e6-b6d0-65925d95b386","Type":"ContainerDied","Data":"ae42cb848b6abb1114109e290a469a63b98879d1b154d0d9c9347879c35f3cc0"}
Nov 03 14:44:44.102166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.102145    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f" event={"ID":"b6cfc825-e1b1-46e6-b6d0-65925d95b386","Type":"ContainerStarted","Data":"e2a7178d45984c1a972e374c01956e6eb27f3e44973178f717c441299088226b"}
Nov 03 14:44:44.143294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:44.143252    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd" podStartSLOduration=1.143218433 podStartE2EDuration="1.143218433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:44.141946657 +0000 UTC m=+1962.191977841" watchObservedRunningTime="2026-11-03 14:44:44.143218433 +0000 UTC m=+1962.193249616"
Nov 03 14:44:45.114717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.114680    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:45.237082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.237054    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:45.245354 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.245333    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc\" (UniqueName: \"kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc\") pod \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") "
Nov 03 14:44:45.245483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.245376    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca\") pod \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") "
Nov 03 14:44:45.245483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.245434    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca\") pod \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\" (UID: \"b6cfc825-e1b1-46e6-b6d0-65925d95b386\") "
Nov 03 14:44:45.245790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.245754    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca" pod "b6cfc825-e1b1-46e6-b6d0-65925d95b386" (UID: "b6cfc825-e1b1-46e6-b6d0-65925d95b386"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:45.245927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.245841    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:45.247398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.247369    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc" pod "b6cfc825-e1b1-46e6-b6d0-65925d95b386" (UID: "b6cfc825-e1b1-46e6-b6d0-65925d95b386"). InnerVolumeSpecName "kube-api-access-68hkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:45.247507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.247491    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca" pod "b6cfc825-e1b1-46e6-b6d0-65925d95b386" (UID: "b6cfc825-e1b1-46e6-b6d0-65925d95b386"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:44:45.346663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.346622    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68hkc\" (UniqueName: \"kubernetes.io/projected/b6cfc825-e1b1-46e6-b6d0-65925d95b386-kube-api-access-68hkc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:45.346663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:45.346663    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6cfc825-e1b1-46e6-b6d0-65925d95b386-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:46.115047 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:46.115020    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"
Nov 03 14:44:46.115440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:46.115054    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f" event={"ID":"b6cfc825-e1b1-46e6-b6d0-65925d95b386","Type":"ContainerDied","Data":"e2a7178d45984c1a972e374c01956e6eb27f3e44973178f717c441299088226b"}
Nov 03 14:44:46.115440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:46.115098    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2a7178d45984c1a972e374c01956e6eb27f3e44973178f717c441299088226b"
Nov 03 14:44:47.095647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:47.095607    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:48.093821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:48.093786    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:49.181309 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.181276    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:49.804176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.804138    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"]
Nov 03 14:44:49.804908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.804881    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b6cfc825-e1b1-46e6-b6d0-65925d95b386" containerName="revoke-keys"
Nov 03 14:44:49.804908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.804907    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cfc825-e1b1-46e6-b6d0-65925d95b386" containerName="revoke-keys"
Nov 03 14:44:49.805088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.805018    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cfc825-e1b1-46e6-b6d0-65925d95b386" containerName="revoke-keys"
Nov 03 14:44:49.808977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.808956    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.830792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.830770    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"]
Nov 03 14:44:49.888209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888179    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888219    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888265    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888420    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888463    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrs5\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888553    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888593    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888671    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.888805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.888724    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990078    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990122    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990270    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990318    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990367    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990389    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrs5\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990412    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990453    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990520    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990594    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990612    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990724    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.990899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.990850    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.991203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.991186    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.992587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.992558    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:49.992988 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:49.992967    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:50.011147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.011119    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:50.011523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.011506    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrs5\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5\") pod \"e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") " pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:50.123477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.123388    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:50.441455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.441427    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"]
Nov 03 14:44:50.442780 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:50.442751    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c3947e_e956_43af_b11a_966ae8ee494f.slice/crio-57c1c25a6e4547967a9cd7d050bfef620426cdffad37ad974215968726b3c478 WatchSource:0}: Error finding container 57c1c25a6e4547967a9cd7d050bfef620426cdffad37ad974215968726b3c478: Status 404 returned error can't find the container with id 57c1c25a6e4547967a9cd7d050bfef620426cdffad37ad974215968726b3c478
Nov 03 14:44:50.445013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.444982    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:44:50.445090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.445045    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:44:50.445090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:50.445073    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:44:51.157528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:51.157485    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" event={"ID":"08c3947e-e956-43af-b11a-966ae8ee494f","Type":"ContainerStarted","Data":"9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87"}
Nov 03 14:44:51.157765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:51.157556    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" event={"ID":"08c3947e-e956-43af-b11a-966ae8ee494f","Type":"ContainerStarted","Data":"57c1c25a6e4547967a9cd7d050bfef620426cdffad37ad974215968726b3c478"}
Nov 03 14:44:51.194802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:51.194774    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:51.238982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:51.238929    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podStartSLOduration=2.238916017 podStartE2EDuration="2.238916017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:51.225901 +0000 UTC m=+1969.275932184" watchObservedRunningTime="2026-11-03 14:44:51.238916017 +0000 UTC m=+1969.288947198"
Nov 03 14:44:52.125405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:52.125374    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:52.126985 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:52.126956    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 14:44:52.127135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:52.127012    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 14:44:53.125592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.125559    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 14:44:53.126038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.125611    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 14:44:53.187716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.187682    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:53.286377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.286348    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"]
Nov 03 14:44:53.290161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.290143    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.330096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.328508    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"]
Nov 03 14:44:53.330096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.328784    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:44:53.330096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.328926    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:44:53.330096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.328975    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:44:53.330096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.329030    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:44:53.347133 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.347104    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.347272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.347147    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fvq\" (UniqueName: \"kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.347338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.347324    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.448503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.448475    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.448676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.448603    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.448676 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.448641    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvq\" (UniqueName: \"kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.449253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.449212    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.450804 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.450785    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.469158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.469135    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fvq\" (UniqueName: \"kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq\") pod \"maas-api-revoke-sub-2ad6b4f5-zqrbz\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.632587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.632551    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:53.935684 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:53.935654    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"]
Nov 03 14:44:53.937317 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:44:53.937292    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af9f81e_594b_402c_9094_7756765523b0.slice/crio-a0cf2789f44abd3512fd552f9e6c780ce62418b155dcef01568f0d2fd4247296 WatchSource:0}: Error finding container a0cf2789f44abd3512fd552f9e6c780ce62418b155dcef01568f0d2fd4247296: Status 404 returned error can't find the container with id a0cf2789f44abd3512fd552f9e6c780ce62418b155dcef01568f0d2fd4247296
Nov 03 14:44:54.128253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:54.126126    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 14:44:54.128253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:54.126182    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 14:44:54.128253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:54.127408    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:44:54.176224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:54.176176    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz" event={"ID":"0af9f81e-594b-402c-9094-7756765523b0","Type":"ContainerStarted","Data":"a0cf2789f44abd3512fd552f9e6c780ce62418b155dcef01568f0d2fd4247296"}
Nov 03 14:44:54.229367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:54.229332    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"]
Nov 03 14:44:55.125598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.125563    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 14:44:55.125790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.125615    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 14:44:55.126094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.126072    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:44:55.183203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.183102    6481 generic.go:356] "Generic (PLEG): container finished" podID="0af9f81e-594b-402c-9094-7756765523b0" containerID="70b878d8dcb8ca6ca9d18f47327d67b6b3eade1cdf73520e65e29783e2c91a24" exitCode=0
Nov 03 14:44:55.183203 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.183136    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz" event={"ID":"0af9f81e-594b-402c-9094-7756765523b0","Type":"ContainerDied","Data":"70b878d8dcb8ca6ca9d18f47327d67b6b3eade1cdf73520e65e29783e2c91a24"}
Nov 03 14:44:55.216252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.206494    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:55.230938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.230906    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"]
Nov 03 14:44:55.459425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:55.459340    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:44:56.125538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.125502    6481 patch_prober.go:29] interesting pod/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused" start-of-body=
Nov 03 14:44:56.125696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.125562    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.215:15021/healthz/ready\": dial tcp 10.133.0.215:15021: connect: connection refused"
Nov 03 14:44:56.355281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.355222    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:56.382959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.382888    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca\") pod \"0af9f81e-594b-402c-9094-7756765523b0\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") "
Nov 03 14:44:56.382959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.382923    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca\") pod \"0af9f81e-594b-402c-9094-7756765523b0\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") "
Nov 03 14:44:56.383184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.383017    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq\" (UniqueName: \"kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq\") pod \"0af9f81e-594b-402c-9094-7756765523b0\" (UID: \"0af9f81e-594b-402c-9094-7756765523b0\") "
Nov 03 14:44:56.383339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.383317    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca" pod "0af9f81e-594b-402c-9094-7756765523b0" (UID: "0af9f81e-594b-402c-9094-7756765523b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:56.385009 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.384979    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca" pod "0af9f81e-594b-402c-9094-7756765523b0" (UID: "0af9f81e-594b-402c-9094-7756765523b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:44:56.385095 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.385052    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq" pod "0af9f81e-594b-402c-9094-7756765523b0" (UID: "0af9f81e-594b-402c-9094-7756765523b0"). InnerVolumeSpecName "kube-api-access-97fvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:56.484055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.484020    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af9f81e-594b-402c-9094-7756765523b0-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:56.484055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.484050    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af9f81e-594b-402c-9094-7756765523b0-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:56.484055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:56.484060    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97fvq\" (UniqueName: \"kubernetes.io/projected/0af9f81e-594b-402c-9094-7756765523b0-kube-api-access-97fvq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:57.129971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.129928    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:57.130920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.130894    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:57.131771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.131744    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:44:57.178810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.178779    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:57.196428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.196399    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"
Nov 03 14:44:57.196428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.196412    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz" event={"ID":"0af9f81e-594b-402c-9094-7756765523b0","Type":"ContainerDied","Data":"a0cf2789f44abd3512fd552f9e6c780ce62418b155dcef01568f0d2fd4247296"}
Nov 03 14:44:57.196650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:57.196454    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0cf2789f44abd3512fd552f9e6c780ce62418b155dcef01568f0d2fd4247296"
Nov 03 14:44:57.985981 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:44:57.985943    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:44:59.185286 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.185249    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:44:59.230222 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.230188    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9" podUID="1512f5d3-14c5-4d0f-ae08-4a443798d18f" containerName="payload-processing" containerID="cri-o://50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5" gracePeriod=25
Nov 03 14:44:59.477715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.477688    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:44:59.517024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.516999    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials\") pod \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") "
Nov 03 14:44:59.517153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.517039    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume\") pod \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") "
Nov 03 14:44:59.517153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.517065    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq\") pod \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") "
Nov 03 14:44:59.517153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.517105    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume\") pod \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\" (UID: \"1512f5d3-14c5-4d0f-ae08-4a443798d18f\") "
Nov 03 14:44:59.517728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.517689    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume" pod "1512f5d3-14c5-4d0f-ae08-4a443798d18f" (UID: "1512f5d3-14c5-4d0f-ae08-4a443798d18f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:59.517848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.517790    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:59.518270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.518209    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume" pod "1512f5d3-14c5-4d0f-ae08-4a443798d18f" (UID: "1512f5d3-14c5-4d0f-ae08-4a443798d18f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:44:59.519956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.519931    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq" pod "1512f5d3-14c5-4d0f-ae08-4a443798d18f" (UID: "1512f5d3-14c5-4d0f-ae08-4a443798d18f"). InnerVolumeSpecName "kube-api-access-6jzbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:59.520934 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.520910    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials" pod "1512f5d3-14c5-4d0f-ae08-4a443798d18f" (UID: "1512f5d3-14c5-4d0f-ae08-4a443798d18f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:44:59.618856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.618826    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:59.618856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.618853    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/1512f5d3-14c5-4d0f-ae08-4a443798d18f-kube-api-access-6jzbq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:44:59.618856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:44:59.618862    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1512f5d3-14c5-4d0f-ae08-4a443798d18f-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:00.155596 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.155562    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"]
Nov 03 14:45:00.156194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.156170    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1512f5d3-14c5-4d0f-ae08-4a443798d18f" containerName="payload-processing"
Nov 03 14:45:00.156194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.156192    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1512f5d3-14c5-4d0f-ae08-4a443798d18f" containerName="payload-processing"
Nov 03 14:45:00.156478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.156242    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0af9f81e-594b-402c-9094-7756765523b0" containerName="revoke-keys"
Nov 03 14:45:00.156478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.156249    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af9f81e-594b-402c-9094-7756765523b0" containerName="revoke-keys"
Nov 03 14:45:00.156478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.156331    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1512f5d3-14c5-4d0f-ae08-4a443798d18f" containerName="payload-processing"
Nov 03 14:45:00.156478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.156345    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af9f81e-594b-402c-9094-7756765523b0" containerName="revoke-keys"
Nov 03 14:45:00.159825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.159806    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"]
Nov 03 14:45:00.159950 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.159932    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.163449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.163432    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.181007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.180983    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"]
Nov 03 14:45:00.183743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.183721    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"]
Nov 03 14:45:00.189921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.189693    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:45:00.189921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.189736    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwhgc\""
Nov 03 14:45:00.189921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.189878    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:45:00.216385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.216357    6481 generic.go:356] "Generic (PLEG): container finished" podID="1512f5d3-14c5-4d0f-ae08-4a443798d18f" containerID="50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5" exitCode=0
Nov 03 14:45:00.216519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.216416    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"
Nov 03 14:45:00.216519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.216436    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9" event={"ID":"1512f5d3-14c5-4d0f-ae08-4a443798d18f","Type":"ContainerDied","Data":"50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5"}
Nov 03 14:45:00.216519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.216482    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9" event={"ID":"1512f5d3-14c5-4d0f-ae08-4a443798d18f","Type":"ContainerDied","Data":"a78e30e1d081a69cd8ca51be2123243bb8f454d1bc8989af9e39e0d0e9920f55"}
Nov 03 14:45:00.216519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.216506    6481 scope.go:122] "RemoveContainer" containerID="50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5"
Nov 03 14:45:00.224532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.224495    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9jf\" (UniqueName: \"kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.224644 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.224584    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.224698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.224646    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.224698 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.224675    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh29f\" (UniqueName: \"kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.226015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.225998    6481 scope.go:122] "RemoveContainer" containerID="50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5"
Nov 03 14:45:00.230269 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:00.230239    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5\": container with ID starting with 50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5 not found: ID does not exist" containerID="50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5"
Nov 03 14:45:00.230375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.230278    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5"} err="failed to get container status \"50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5\": rpc error: code = NotFound desc = could not find container \"50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5\": container with ID starting with 50f9789059379a6c318f702139969e7dfaf4ac3aea8f638a51a767d9f0175be5 not found: ID does not exist"
Nov 03 14:45:00.231686 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.231663    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr" podUID="6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" containerName="payload-processing" containerID="cri-o://fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae" gracePeriod=25
Nov 03 14:45:00.246403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.246361    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:00.297825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.297798    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"]
Nov 03 14:45:00.301368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.301348    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-5865jfbq9"]
Nov 03 14:45:00.325438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.325386    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.325438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.325424    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh29f\" (UniqueName: \"kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.325563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.325519    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9jf\" (UniqueName: \"kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.325563 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.325538    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.326100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.326081    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.326141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.326106    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.346505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.346482    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9jf\" (UniqueName: \"kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf\") pod \"maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.346952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.346929    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh29f\" (UniqueName: \"kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f\") pod \"maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.473981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.473958    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:00.480955 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.480929    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:00.495583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.495563    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:45:00.629264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629178    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume\") pod \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") "
Nov 03 14:45:00.629391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629294    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q\") pod \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") "
Nov 03 14:45:00.629391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629348    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials\") pod \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") "
Nov 03 14:45:00.629497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629425    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume\") pod \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\" (UID: \"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065\") "
Nov 03 14:45:00.629798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629762    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume" pod "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" (UID: "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:00.629920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629798    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume" pod "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" (UID: "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:00.629920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629899    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:00.629920 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.629917    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:00.631416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.631387    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q" pod "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" (UID: "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065"). InnerVolumeSpecName "kube-api-access-hbz9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:00.631519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.631468    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials" pod "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" (UID: "6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:00.730789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.730763    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbz9q\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-kube-api-access-hbz9q\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:00.730789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.730787    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:00.775054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.775026    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"]
Nov 03 14:45:00.776098 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:45:00.776072    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68d0305_6712_4c63_9f73_4e6c4a5ecde5.slice/crio-a6601cd4651852406adfab20b073a28cfee167d88c82e403a1aa9102b8e5c2f1 WatchSource:0}: Error finding container a6601cd4651852406adfab20b073a28cfee167d88c82e403a1aa9102b8e5c2f1: Status 404 returned error can't find the container with id a6601cd4651852406adfab20b073a28cfee167d88c82e403a1aa9102b8e5c2f1
Nov 03 14:45:00.801558 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:00.801535    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"]
Nov 03 14:45:00.807314 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:45:00.807292    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad954b63_e848_476a_a944_5140d5755c17.slice/crio-ea28df38fa13eb4812e7840b856e96559e36b83bb6124f8da696e65a62bcca6f WatchSource:0}: Error finding container ea28df38fa13eb4812e7840b856e96559e36b83bb6124f8da696e65a62bcca6f: Status 404 returned error can't find the container with id ea28df38fa13eb4812e7840b856e96559e36b83bb6124f8da696e65a62bcca6f
Nov 03 14:45:01.187977 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.187948    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:01.224442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.224409    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" event={"ID":"ad954b63-e848-476a-a944-5140d5755c17","Type":"ContainerStarted","Data":"e79cdd5d4ff6d3b3972ea61511e5f19ee767e4a4306b9d665534cc92a1603690"}
Nov 03 14:45:01.224994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.224467    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" event={"ID":"ad954b63-e848-476a-a944-5140d5755c17","Type":"ContainerStarted","Data":"ea28df38fa13eb4812e7840b856e96559e36b83bb6124f8da696e65a62bcca6f"}
Nov 03 14:45:01.226156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.226124    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerStarted","Data":"8080ba4416a91d4182681b033117b3c51e8fc7ce0b4e587884112ddd2c8832aa"}
Nov 03 14:45:01.226304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.226174    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerStarted","Data":"a6601cd4651852406adfab20b073a28cfee167d88c82e403a1aa9102b8e5c2f1"}
Nov 03 14:45:01.227544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.227517    6481 generic.go:356] "Generic (PLEG): container finished" podID="6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" containerID="fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae" exitCode=0
Nov 03 14:45:01.227638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.227553    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"
Nov 03 14:45:01.227638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.227596    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr" event={"ID":"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065","Type":"ContainerDied","Data":"fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae"}
Nov 03 14:45:01.227735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.227644    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr" event={"ID":"6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065","Type":"ContainerDied","Data":"6fe35b6cf25f753a4d900aee14d55501b7917ecd04d7bbe3879021ec53217a78"}
Nov 03 14:45:01.227735 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.227673    6481 scope.go:122] "RemoveContainer" containerID="fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae"
Nov 03 14:45:01.237978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.237762    6481 scope.go:122] "RemoveContainer" containerID="fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae"
Nov 03 14:45:01.238096 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:01.238074    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae\": container with ID starting with fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae not found: ID does not exist" containerID="fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae"
Nov 03 14:45:01.238142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.238106    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae"} err="failed to get container status \"fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae\": rpc error: code = NotFound desc = could not find container \"fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae\": container with ID starting with fb91a826170796203600ef7644e24c20c158d943379d12714007cd4846d212ae not found: ID does not exist"
Nov 03 14:45:01.266938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.266862    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" podStartSLOduration=1.266827743 podStartE2EDuration="1.266827743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:01.266418519 +0000 UTC m=+1979.316449705" watchObservedRunningTime="2026-11-03 14:45:01.266827743 +0000 UTC m=+1979.316858927"
Nov 03 14:45:01.317921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.317876    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" podStartSLOduration=1.317858037 podStartE2EDuration="1.317858037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:01.31753263 +0000 UTC m=+1979.367563813" watchObservedRunningTime="2026-11-03 14:45:01.317858037 +0000 UTC m=+1979.367889224"
Nov 03 14:45:01.357442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.357415    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"]
Nov 03 14:45:01.360014 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:01.359993    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-9cd4gxgqr"]
Nov 03 14:45:02.068081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.068046    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1512f5d3-14c5-4d0f-ae08-4a443798d18f" path="/var/lib/kubelet/pods/1512f5d3-14c5-4d0f-ae08-4a443798d18f/volumes"
Nov 03 14:45:02.068565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.068543    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" path="/var/lib/kubelet/pods/6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065/volumes"
Nov 03 14:45:02.236506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.236468    6481 generic.go:356] "Generic (PLEG): container finished" podID="ad954b63-e848-476a-a944-5140d5755c17" containerID="e79cdd5d4ff6d3b3972ea61511e5f19ee767e4a4306b9d665534cc92a1603690" exitCode=60
Nov 03 14:45:02.236984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.236524    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" event={"ID":"ad954b63-e848-476a-a944-5140d5755c17","Type":"ContainerDied","Data":"e79cdd5d4ff6d3b3972ea61511e5f19ee767e4a4306b9d665534cc92a1603690"}
Nov 03 14:45:02.236984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.236802    6481 scope.go:122] "RemoveContainer" containerID="e79cdd5d4ff6d3b3972ea61511e5f19ee767e4a4306b9d665534cc92a1603690"
Nov 03 14:45:02.239679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.239654    6481 generic.go:356] "Generic (PLEG): container finished" podID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerID="8080ba4416a91d4182681b033117b3c51e8fc7ce0b4e587884112ddd2c8832aa" exitCode=60
Nov 03 14:45:02.239831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.239683    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerDied","Data":"8080ba4416a91d4182681b033117b3c51e8fc7ce0b4e587884112ddd2c8832aa"}
Nov 03 14:45:02.240068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:02.239964    6481 scope.go:122] "RemoveContainer" containerID="8080ba4416a91d4182681b033117b3c51e8fc7ce0b4e587884112ddd2c8832aa"
Nov 03 14:45:03.202699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.202589    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:03.247599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.247566    6481 generic.go:356] "Generic (PLEG): container finished" podID="ad954b63-e848-476a-a944-5140d5755c17" containerID="8d6d1e9cd9e002e0638561360045ff4594e66f47a4be0783748e527ba969d950" exitCode=60
Nov 03 14:45:03.248060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.247646    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" event={"ID":"ad954b63-e848-476a-a944-5140d5755c17","Type":"ContainerDied","Data":"8d6d1e9cd9e002e0638561360045ff4594e66f47a4be0783748e527ba969d950"}
Nov 03 14:45:03.248060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.247701    6481 scope.go:122] "RemoveContainer" containerID="e79cdd5d4ff6d3b3972ea61511e5f19ee767e4a4306b9d665534cc92a1603690"
Nov 03 14:45:03.248060 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.248037    6481 scope.go:122] "RemoveContainer" containerID="8d6d1e9cd9e002e0638561360045ff4594e66f47a4be0783748e527ba969d950"
Nov 03 14:45:03.248359 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:03.248340    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp_odh-ai-gateway-infra(ad954b63-e848-476a-a944-5140d5755c17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" podUID="ad954b63-e848-476a-a944-5140d5755c17"
Nov 03 14:45:03.249584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.249565    6481 generic.go:356] "Generic (PLEG): container finished" podID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerID="a18d27eb147f01f8eafb4d34f1fcf14b731c4abbe1020d6c50b6c60ee26d69ca" exitCode=60
Nov 03 14:45:03.249668 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.249600    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerDied","Data":"a18d27eb147f01f8eafb4d34f1fcf14b731c4abbe1020d6c50b6c60ee26d69ca"}
Nov 03 14:45:03.249848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.249835    6481 scope.go:122] "RemoveContainer" containerID="a18d27eb147f01f8eafb4d34f1fcf14b731c4abbe1020d6c50b6c60ee26d69ca"
Nov 03 14:45:03.250072 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:03.250056    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8_odh-ai-gateway-infra(d68d0305-6712-4c63-9f73-4e6c4a5ecde5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5"
Nov 03 14:45:03.260146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:03.260123    6481 scope.go:122] "RemoveContainer" containerID="8080ba4416a91d4182681b033117b3c51e8fc7ce0b4e587884112ddd2c8832aa"
Nov 03 14:45:04.256631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:04.256599    6481 scope.go:122] "RemoveContainer" containerID="8d6d1e9cd9e002e0638561360045ff4594e66f47a4be0783748e527ba969d950"
Nov 03 14:45:04.257092 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:04.256872    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp_odh-ai-gateway-infra(ad954b63-e848-476a-a944-5140d5755c17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" podUID="ad954b63-e848-476a-a944-5140d5755c17"
Nov 03 14:45:04.258169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:04.258152    6481 scope.go:122] "RemoveContainer" containerID="a18d27eb147f01f8eafb4d34f1fcf14b731c4abbe1020d6c50b6c60ee26d69ca"
Nov 03 14:45:04.258392 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:04.258373    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8_odh-ai-gateway-infra(d68d0305-6712-4c63-9f73-4e6c4a5ecde5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5"
Nov 03 14:45:05.009371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:05.009337    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"]
Nov 03 14:45:05.178345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:05.178308    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:05.392302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:05.385206    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:05.821613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:05.821586    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:06.052370 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:06.052337    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"]
Nov 03 14:45:06.212902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:06.212877    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:07.071344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:07.071313    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:45:07.208407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:07.208377    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:07.817158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:07.817127    6481 scope.go:122] "RemoveContainer" containerID="2ac1734fe87d555818acc14c4a92da398ac7a1fdce0b186ec38ca37c1cf3aa33"
Nov 03 14:45:07.826595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:07.826575    6481 scope.go:122] "RemoveContainer" containerID="eb0046464bf537ef6490c06dc2baa176b1faac77fb7377b912d8307c87f7e145"
Nov 03 14:45:09.315650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:09.315623    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:10.010129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.010073    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp" podUID="e5517347-66e7-402c-b301-1e29c5dcbe8b" containerName="payload-processing" containerID="cri-o://bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06" gracePeriod=25
Nov 03 14:45:10.263793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.263741    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:45:10.294808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.294778    6481 generic.go:356] "Generic (PLEG): container finished" podID="e5517347-66e7-402c-b301-1e29c5dcbe8b" containerID="bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06" exitCode=0
Nov 03 14:45:10.294931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.294819    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"
Nov 03 14:45:10.294931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.294868    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp" event={"ID":"e5517347-66e7-402c-b301-1e29c5dcbe8b","Type":"ContainerDied","Data":"bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06"}
Nov 03 14:45:10.294931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.294921    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp" event={"ID":"e5517347-66e7-402c-b301-1e29c5dcbe8b","Type":"ContainerDied","Data":"54788238878b56c8de972a17f8a2397cca56b5d13d4a485bfa05d26c31788b3f"}
Nov 03 14:45:10.295055 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.294944    6481 scope.go:122] "RemoveContainer" containerID="bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06"
Nov 03 14:45:10.303769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.303753    6481 scope.go:122] "RemoveContainer" containerID="bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06"
Nov 03 14:45:10.304018 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:10.304001    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06\": container with ID starting with bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06 not found: ID does not exist" containerID="bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06"
Nov 03 14:45:10.304094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.304026    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06"} err="failed to get container status \"bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06\": rpc error: code = NotFound desc = could not find container \"bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06\": container with ID starting with bf1d57edd9cddf83ec8468559f6b3bce8fa01c3cfd819ec33107d1bce1beff06 not found: ID does not exist"
Nov 03 14:45:10.356717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.356690    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume\") pod \"e5517347-66e7-402c-b301-1e29c5dcbe8b\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") "
Nov 03 14:45:10.357112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.356789    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq\") pod \"e5517347-66e7-402c-b301-1e29c5dcbe8b\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") "
Nov 03 14:45:10.357112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.356846    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume\") pod \"e5517347-66e7-402c-b301-1e29c5dcbe8b\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") "
Nov 03 14:45:10.357112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.356907    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials\") pod \"e5517347-66e7-402c-b301-1e29c5dcbe8b\" (UID: \"e5517347-66e7-402c-b301-1e29c5dcbe8b\") "
Nov 03 14:45:10.357112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.357059    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume" pod "e5517347-66e7-402c-b301-1e29c5dcbe8b" (UID: "e5517347-66e7-402c-b301-1e29c5dcbe8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:10.357345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.357198    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume" pod "e5517347-66e7-402c-b301-1e29c5dcbe8b" (UID: "e5517347-66e7-402c-b301-1e29c5dcbe8b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:10.357345 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.357294    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:10.358919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.358894    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials" pod "e5517347-66e7-402c-b301-1e29c5dcbe8b" (UID: "e5517347-66e7-402c-b301-1e29c5dcbe8b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:10.358999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.358919    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq" pod "e5517347-66e7-402c-b301-1e29c5dcbe8b" (UID: "e5517347-66e7-402c-b301-1e29c5dcbe8b"). InnerVolumeSpecName "kube-api-access-wmmpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:10.458053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.458029    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:10.458053 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.458050    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmmpq\" (UniqueName: \"kubernetes.io/projected/e5517347-66e7-402c-b301-1e29c5dcbe8b-kube-api-access-wmmpq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:10.458186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.458060    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5517347-66e7-402c-b301-1e29c5dcbe8b-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:10.641015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.640987    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"]
Nov 03 14:45:10.647942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:10.647922    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-331c9e/payload-processing-external-model-e2e-extmod-a-331c9e-758f6r8gp"]
Nov 03 14:45:11.053533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.053475    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd" podUID="f90933fd-b96a-4e6e-bc8e-f2accbc0df53" containerName="payload-processing" containerID="cri-o://503738e50797b66e0ccdcaa2c10a031af3481135347768b02b155d62bf3e75ed" gracePeriod=25
Nov 03 14:45:11.303159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.303119    6481 generic.go:356] "Generic (PLEG): container finished" podID="f90933fd-b96a-4e6e-bc8e-f2accbc0df53" containerID="503738e50797b66e0ccdcaa2c10a031af3481135347768b02b155d62bf3e75ed" exitCode=0
Nov 03 14:45:11.303159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.303182    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd" event={"ID":"f90933fd-b96a-4e6e-bc8e-f2accbc0df53","Type":"ContainerDied","Data":"503738e50797b66e0ccdcaa2c10a031af3481135347768b02b155d62bf3e75ed"}
Nov 03 14:45:11.313011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.312987    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:11.328656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.328635    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:45:11.468949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.468921    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn\") pod \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") "
Nov 03 14:45:11.468949 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.468951    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume\") pod \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") "
Nov 03 14:45:11.469455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.468987    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume\") pod \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") "
Nov 03 14:45:11.469455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.469140    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials\") pod \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\" (UID: \"f90933fd-b96a-4e6e-bc8e-f2accbc0df53\") "
Nov 03 14:45:11.469455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.469347    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume" pod "f90933fd-b96a-4e6e-bc8e-f2accbc0df53" (UID: "f90933fd-b96a-4e6e-bc8e-f2accbc0df53"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:11.469455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.469383    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume" pod "f90933fd-b96a-4e6e-bc8e-f2accbc0df53" (UID: "f90933fd-b96a-4e6e-bc8e-f2accbc0df53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:11.469626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.469558    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:11.469626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.469571    6481 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-routing-overlay-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:11.471013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.470993    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn" pod "f90933fd-b96a-4e6e-bc8e-f2accbc0df53" (UID: "f90933fd-b96a-4e6e-bc8e-f2accbc0df53"). InnerVolumeSpecName "kube-api-access-rpnjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:11.471097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.471060    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials" pod "f90933fd-b96a-4e6e-bc8e-f2accbc0df53" (UID: "f90933fd-b96a-4e6e-bc8e-f2accbc0df53"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:11.570793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.570719    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpnjn\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-kube-api-access-rpnjn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:11.570793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:11.570747    6481 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f90933fd-b96a-4e6e-bc8e-f2accbc0df53-provider-credentials\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:12.066547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:12.066516    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5517347-66e7-402c-b301-1e29c5dcbe8b" path="/var/lib/kubelet/pods/e5517347-66e7-402c-b301-1e29c5dcbe8b/volumes"
Nov 03 14:45:12.309961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:12.309928    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"
Nov 03 14:45:12.310157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:12.309973    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd" event={"ID":"f90933fd-b96a-4e6e-bc8e-f2accbc0df53","Type":"ContainerDied","Data":"1184968523dfe4bd85d5b588879b5b64619a6f7a378d3500959100cd66f608b3"}
Nov 03 14:45:12.310157 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:12.310044    6481 scope.go:122] "RemoveContainer" containerID="503738e50797b66e0ccdcaa2c10a031af3481135347768b02b155d62bf3e75ed"
Nov 03 14:45:12.363255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:12.363219    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"]
Nov 03 14:45:12.374689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:12.374662    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b4361f/payload-processing-external-model-e2e-extmod-b-b4361f-b879b2twd"]
Nov 03 14:45:13.302833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:13.302757    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:13.782903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:13.782870    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:13.989726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:13.989693    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:14.060272 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:14.060184    6481 scope.go:122] "RemoveContainer" containerID="8d6d1e9cd9e002e0638561360045ff4594e66f47a4be0783748e527ba969d950"
Nov 03 14:45:14.066557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:14.066530    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f90933fd-b96a-4e6e-bc8e-f2accbc0df53" path="/var/lib/kubelet/pods/f90933fd-b96a-4e6e-bc8e-f2accbc0df53/volumes"
Nov 03 14:45:14.284844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:14.284814    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:15.295326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:15.295245    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:15.329069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:15.329037    6481 generic.go:356] "Generic (PLEG): container finished" podID="ad954b63-e848-476a-a944-5140d5755c17" containerID="3fff148e2416a77a68ef683684af3e2b366773d39b7dbc390db33dd3069c5950" exitCode=60
Nov 03 14:45:15.329221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:15.329117    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" event={"ID":"ad954b63-e848-476a-a944-5140d5755c17","Type":"ContainerDied","Data":"3fff148e2416a77a68ef683684af3e2b366773d39b7dbc390db33dd3069c5950"}
Nov 03 14:45:15.329221 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:15.329205    6481 scope.go:122] "RemoveContainer" containerID="8d6d1e9cd9e002e0638561360045ff4594e66f47a4be0783748e527ba969d950"
Nov 03 14:45:15.329577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:15.329556    6481 scope.go:122] "RemoveContainer" containerID="3fff148e2416a77a68ef683684af3e2b366773d39b7dbc390db33dd3069c5950"
Nov 03 14:45:15.329872 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:15.329841    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp_odh-ai-gateway-infra(ad954b63-e848-476a-a944-5140d5755c17)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" podUID="ad954b63-e848-476a-a944-5140d5755c17"
Nov 03 14:45:16.060187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.060149    6481 scope.go:122] "RemoveContainer" containerID="a18d27eb147f01f8eafb4d34f1fcf14b731c4abbe1020d6c50b6c60ee26d69ca"
Nov 03 14:45:16.338448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.338420    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerStarted","Data":"f34799b4a05a0e2ed3d32e158940caceca06d0afa9675e5439c9c8659ebd626f"}
Nov 03 14:45:16.386207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.386174    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"]
Nov 03 14:45:16.513328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.513304    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:16.624025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.623955    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f\" (UniqueName: \"kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f\") pod \"ad954b63-e848-476a-a944-5140d5755c17\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") "
Nov 03 14:45:16.624166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.624032    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca\") pod \"ad954b63-e848-476a-a944-5140d5755c17\" (UID: \"ad954b63-e848-476a-a944-5140d5755c17\") "
Nov 03 14:45:16.624414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.624389    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca" pod "ad954b63-e848-476a-a944-5140d5755c17" (UID: "ad954b63-e848-476a-a944-5140d5755c17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:16.624544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.624523    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad954b63-e848-476a-a944-5140d5755c17-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:16.626039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.626013    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f" pod "ad954b63-e848-476a-a944-5140d5755c17" (UID: "ad954b63-e848-476a-a944-5140d5755c17"). InnerVolumeSpecName "kube-api-access-zh29f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:16.725332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:16.725307    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh29f\" (UniqueName: \"kubernetes.io/projected/ad954b63-e848-476a-a944-5140d5755c17-kube-api-access-zh29f\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:17.315504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.315468    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:17.344601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.344574    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"
Nov 03 14:45:17.345034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.344573    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp" event={"ID":"ad954b63-e848-476a-a944-5140d5755c17","Type":"ContainerDied","Data":"ea28df38fa13eb4812e7840b856e96559e36b83bb6124f8da696e65a62bcca6f"}
Nov 03 14:45:17.345034 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.344700    6481 scope.go:122] "RemoveContainer" containerID="3fff148e2416a77a68ef683684af3e2b366773d39b7dbc390db33dd3069c5950"
Nov 03 14:45:17.346597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.346566    6481 generic.go:356] "Generic (PLEG): container finished" podID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerID="f34799b4a05a0e2ed3d32e158940caceca06d0afa9675e5439c9c8659ebd626f" exitCode=60
Nov 03 14:45:17.346740 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.346593    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerDied","Data":"f34799b4a05a0e2ed3d32e158940caceca06d0afa9675e5439c9c8659ebd626f"}
Nov 03 14:45:17.346945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.346928    6481 scope.go:122] "RemoveContainer" containerID="f34799b4a05a0e2ed3d32e158940caceca06d0afa9675e5439c9c8659ebd626f"
Nov 03 14:45:17.347162 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:17.347141    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8_odh-ai-gateway-infra(d68d0305-6712-4c63-9f73-4e6c4a5ecde5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5"
Nov 03 14:45:17.354249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.354214    6481 scope.go:122] "RemoveContainer" containerID="a18d27eb147f01f8eafb4d34f1fcf14b731c4abbe1020d6c50b6c60ee26d69ca"
Nov 03 14:45:17.400953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.400932    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"]
Nov 03 14:45:17.504433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.504409    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"]
Nov 03 14:45:17.510377 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:17.510351    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-331c9e-29895285-8cmjp"]
Nov 03 14:45:18.067003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.066974    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad954b63-e848-476a-a944-5140d5755c17" path="/var/lib/kubelet/pods/ad954b63-e848-476a-a944-5140d5755c17/volumes"
Nov 03 14:45:18.491847 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.491824    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:18.542536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.542504    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf\" (UniqueName: \"kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf\") pod \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") "
Nov 03 14:45:18.542691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.542626    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca\") pod \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\" (UID: \"d68d0305-6712-4c63-9f73-4e6c4a5ecde5\") "
Nov 03 14:45:18.543041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.543012    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca" pod "d68d0305-6712-4c63-9f73-4e6c4a5ecde5" (UID: "d68d0305-6712-4c63-9f73-4e6c4a5ecde5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:18.544757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.544731    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf" pod "d68d0305-6712-4c63-9f73-4e6c4a5ecde5" (UID: "d68d0305-6712-4c63-9f73-4e6c4a5ecde5"). InnerVolumeSpecName "kube-api-access-nx9jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:18.643548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.643471    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9jf\" (UniqueName: \"kubernetes.io/projected/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-kube-api-access-nx9jf\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:18.643548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:18.643504    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d68d0305-6712-4c63-9f73-4e6c4a5ecde5-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:19.364948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:19.364920    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"
Nov 03 14:45:19.364948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:19.364935    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8" event={"ID":"d68d0305-6712-4c63-9f73-4e6c4a5ecde5","Type":"ContainerDied","Data":"a6601cd4651852406adfab20b073a28cfee167d88c82e403a1aa9102b8e5c2f1"}
Nov 03 14:45:19.365185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:19.364992    6481 scope.go:122] "RemoveContainer" containerID="f34799b4a05a0e2ed3d32e158940caceca06d0afa9675e5439c9c8659ebd626f"
Nov 03 14:45:19.390722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:19.390690    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:19.418173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:19.418146    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"]
Nov 03 14:45:19.424454 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:19.424433    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-b4361f-29895285-z8gz8"]
Nov 03 14:45:20.068043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:20.068000    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" path="/var/lib/kubelet/pods/d68d0305-6712-4c63-9f73-4e6c4a5ecde5/volumes"
Nov 03 14:45:20.078199 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:20.078169    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"]
Nov 03 14:45:20.116364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:20.116338    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"]
Nov 03 14:45:20.324940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:20.324826    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:20.538555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:20.538516    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:21.493925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:21.493893    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:23.113128 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:23.113090    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:23.318627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:23.318597    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:23.928300 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:23.928265    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:25.079291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:25.079253    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing" containerID="cri-o://4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4" gracePeriod=25
Nov 03 14:45:25.116964 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:25.116931    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing" containerID="cri-o://49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a" gracePeriod=25
Nov 03 14:45:25.228480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:25.228443    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:45:25.228480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:25.228457    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:45:25.501482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:25.501445    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:45:25.956182 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:25.956139    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:27.877582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:27.877552    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:27.984493 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:27.984458    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:45:29.886287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:29.886254    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:30.564786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:30.564749    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:30.787719 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:30.787685    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:31.380172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:31.380138    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:31.693562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:31.693524    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"]
Nov 03 14:45:31.693770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:31.693746    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy" containerID="cri-o://9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87" gracePeriod=30
Nov 03 14:45:31.705269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:31.705207    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"]
Nov 03 14:45:31.705458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:31.705435    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy" containerID="cri-o://15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9" gracePeriod=30
Nov 03 14:45:31.936344 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:31.936309    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:32.160389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:32.160354    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:32.589038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:32.588950    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:33.983317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:33.983290    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:35.228011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:35.227976    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:45:35.228429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:35.227988    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:45:35.936980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:35.936946    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:36.951022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:36.950999    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:45:36.985729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:36.985707    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:45:37.033387 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033359    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.033584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033458    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.033584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033516    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.033584 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033549    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.033762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033600    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.033762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033640    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.033762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033645    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.033762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033671    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.033762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033705    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033761    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033805    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033837    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033878    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033908    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033917    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.034045 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033978    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034062    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034100    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034152    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket\") pod \"08c3947e-e956-43af-b11a-966ae8ee494f\" (UID: \"08c3947e-e956-43af-b11a-966ae8ee494f\") "
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034201    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data\") pod \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\" (UID: \"aef3669f-7f28-45a7-9e72-7b705e2bc8d5\") "
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.033929    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034111    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034291    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.034442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034324    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034445    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034484    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034643    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034760    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034779    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034823    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034837    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08c3947e-e956-43af-b11a-966ae8ee494f-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034852    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034865    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034878    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034892    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.034904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034905    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.035590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.034848    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.036427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.036378    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:37.037064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.037039    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.037064 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.037060    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:45:37.037293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.037042    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:37.037293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.037281    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:37.037445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.037351    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "kube-api-access-klrs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:37.037687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.037658    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9" pod "aef3669f-7f28-45a7-9e72-7b705e2bc8d5" (UID: "aef3669f-7f28-45a7-9e72-7b705e2bc8d5"). InnerVolumeSpecName "kube-api-access-m48s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:37.038530 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.038512    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo" pod "08c3947e-e956-43af-b11a-966ae8ee494f" (UID: "08c3947e-e956-43af-b11a-966ae8ee494f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:45:37.135457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135427    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m48s9\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-kube-api-access-m48s9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135454    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klrs5\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-kube-api-access-klrs5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135468    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08c3947e-e956-43af-b11a-966ae8ee494f-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135477    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135485    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08c3947e-e956-43af-b11a-966ae8ee494f-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135493    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135500    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135508    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08c3947e-e956-43af-b11a-966ae8ee494f-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.135637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.135516    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aef3669f-7f28-45a7-9e72-7b705e2bc8d5-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:37.466067 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.466032    6481 generic.go:356] "Generic (PLEG): container finished" podID="08c3947e-e956-43af-b11a-966ae8ee494f" containerID="9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87" exitCode=0
Nov 03 14:45:37.466254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.466083    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"
Nov 03 14:45:37.466254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.466121    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" event={"ID":"08c3947e-e956-43af-b11a-966ae8ee494f","Type":"ContainerDied","Data":"9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87"}
Nov 03 14:45:37.466254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.466175    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9" event={"ID":"08c3947e-e956-43af-b11a-966ae8ee494f","Type":"ContainerDied","Data":"57c1c25a6e4547967a9cd7d050bfef620426cdffad37ad974215968726b3c478"}
Nov 03 14:45:37.466254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.466204    6481 scope.go:122] "RemoveContainer" containerID="9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87"
Nov 03 14:45:37.467512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.467495    6481 generic.go:356] "Generic (PLEG): container finished" podID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerID="15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9" exitCode=0
Nov 03 14:45:37.467647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.467549    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"
Nov 03 14:45:37.467647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.467560    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" event={"ID":"aef3669f-7f28-45a7-9e72-7b705e2bc8d5","Type":"ContainerDied","Data":"15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9"}
Nov 03 14:45:37.467647 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.467602    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css" event={"ID":"aef3669f-7f28-45a7-9e72-7b705e2bc8d5","Type":"ContainerDied","Data":"21e8b4bb72f27face994c67de400c553570f942569b21c2bfd7b3cbfb4112656"}
Nov 03 14:45:37.478858 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.478838    6481 scope.go:122] "RemoveContainer" containerID="9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87"
Nov 03 14:45:37.479145 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:37.479128    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87\": container with ID starting with 9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87 not found: ID does not exist" containerID="9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87"
Nov 03 14:45:37.479215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.479155    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87"} err="failed to get container status \"9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87\": rpc error: code = NotFound desc = could not find container \"9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87\": container with ID starting with 9f99b852dbe9e834f61b00e2749c7b4f039998efc1fbfefd7b9e4fcfae394c87 not found: ID does not exist"
Nov 03 14:45:37.479215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.479173    6481 scope.go:122] "RemoveContainer" containerID="15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9"
Nov 03 14:45:37.487980 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.487958    6481 scope.go:122] "RemoveContainer" containerID="15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9"
Nov 03 14:45:37.488292 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:37.488271    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9\": container with ID starting with 15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9 not found: ID does not exist" containerID="15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9"
Nov 03 14:45:37.488388 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.488299    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9"} err="failed to get container status \"15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9\": rpc error: code = NotFound desc = could not find container \"15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9\": container with ID starting with 15e800067f49ad480637738ae482d02416ab45930f6044f0ed6c4b4be07fd4b9 not found: ID does not exist"
Nov 03 14:45:37.519517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.519496    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"]
Nov 03 14:45:37.524607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.524583    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-5l7w9"]
Nov 03 14:45:37.559658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.559629    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"]
Nov 03 14:45:37.564913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.564891    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-331c9e-openshift-default-685b87bb69-l9css"]
Nov 03 14:45:37.888499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:37.888466    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:38.066207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:38.066130    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" path="/var/lib/kubelet/pods/08c3947e-e956-43af-b11a-966ae8ee494f/volumes"
Nov 03 14:45:38.066609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:38.066593    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" path="/var/lib/kubelet/pods/aef3669f-7f28-45a7-9e72-7b705e2bc8d5/volumes"
Nov 03 14:45:39.882907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:39.882873    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:40.134995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:40.134925    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"]
Nov 03 14:45:40.170812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:40.170785    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"]
Nov 03 14:45:40.367361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:40.367324    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:40.569080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:40.569046    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:41.389242 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:41.389208    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:41.665940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:41.665910    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:43.881258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:43.881191    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:45:45.136088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.136057    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerName="payload-processing" containerID="cri-o://f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7" gracePeriod=25
Nov 03 14:45:45.171639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.171607    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" containerName="payload-pre-processing" containerID="cri-o://e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3" gracePeriod=25
Nov 03 14:45:45.227699 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:45:45.227675    6481 logging.go:55] [core] [Channel #3713 SubChannel #3715]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.76:50052", ServerName: "10.133.0.76:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.76:50052: connect: connection refused"
Nov 03 14:45:45.227833 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:45:45.227715    6481 logging.go:55] [core] [Channel #3714 SubChannel #3716]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.77:50052", ServerName: "10.133.0.77:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.77:50052: connect: connection refused"
Nov 03 14:45:45.375943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.375918    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:45:45.397835 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.397816    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:45:45.514077 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.514040    6481 generic.go:356] "Generic (PLEG): container finished" podID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerID="4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4" exitCode=0
Nov 03 14:45:45.514270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.514085    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"
Nov 03 14:45:45.514270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.514128    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" event={"ID":"a8e30eee-5488-49a1-ad18-9b2ca9bd7224","Type":"ContainerDied","Data":"4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4"}
Nov 03 14:45:45.514270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.514180    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" event={"ID":"a8e30eee-5488-49a1-ad18-9b2ca9bd7224","Type":"ContainerDied","Data":"4ecabe8b85b16b0707e26f109440ec62b3593c3183edc2afb527f762e2a94cfc"}
Nov 03 14:45:45.514270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.514211    6481 scope.go:122] "RemoveContainer" containerID="4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4"
Nov 03 14:45:45.515522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.515502    6481 generic.go:356] "Generic (PLEG): container finished" podID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerID="49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a" exitCode=0
Nov 03 14:45:45.515612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.515541    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"
Nov 03 14:45:45.515612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.515572    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" event={"ID":"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5","Type":"ContainerDied","Data":"49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a"}
Nov 03 14:45:45.515831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.515624    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" event={"ID":"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5","Type":"ContainerDied","Data":"f1d01790d7b6cf946bed6e0d2cc1af21eaed0b20ec28791ec18926ba0ec75e37"}
Nov 03 14:45:45.523112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.523097    6481 scope.go:122] "RemoveContainer" containerID="4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4"
Nov 03 14:45:45.523420 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:45.523402    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4\": container with ID starting with 4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4 not found: ID does not exist" containerID="4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4"
Nov 03 14:45:45.523509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.523426    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4"} err="failed to get container status \"4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4\": rpc error: code = NotFound desc = could not find container \"4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4\": container with ID starting with 4c98732e9fd53261b69b197f4dcf158cf85e4aa8db5dd32ca32e32fae22e2cb4 not found: ID does not exist"
Nov 03 14:45:45.523509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.523443    6481 scope.go:122] "RemoveContainer" containerID="49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a"
Nov 03 14:45:45.531608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.531590    6481 scope.go:122] "RemoveContainer" containerID="49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a"
Nov 03 14:45:45.531868 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:45.531850    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a\": container with ID starting with 49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a not found: ID does not exist" containerID="49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a"
Nov 03 14:45:45.531929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.531875    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a"} err="failed to get container status \"49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a\": rpc error: code = NotFound desc = could not find container \"49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a\": container with ID starting with 49253c375f0abc69435c2ff5b4452d89115b9a8debc5182e8fb1d8ca11f24a5a not found: ID does not exist"
Nov 03 14:45:45.532082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532067    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume\") pod \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") "
Nov 03 14:45:45.532120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532098    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume\") pod \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") "
Nov 03 14:45:45.532156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532134    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5\" (UniqueName: \"kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5\") pod \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\" (UID: \"9f26cfa4-d3ab-4973-bfe8-bd9173f524d5\") "
Nov 03 14:45:45.532254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532239    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb\" (UniqueName: \"kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb\") pod \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\" (UID: \"a8e30eee-5488-49a1-ad18-9b2ca9bd7224\") "
Nov 03 14:45:45.532509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532489    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume" pod "a8e30eee-5488-49a1-ad18-9b2ca9bd7224" (UID: "a8e30eee-5488-49a1-ad18-9b2ca9bd7224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:45.532559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532501    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume" pod "9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" (UID: "9f26cfa4-d3ab-4973-bfe8-bd9173f524d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:45.532613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532595    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:45.532648 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.532622    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:45.534134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.534113    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb" pod "a8e30eee-5488-49a1-ad18-9b2ca9bd7224" (UID: "a8e30eee-5488-49a1-ad18-9b2ca9bd7224"). InnerVolumeSpecName "kube-api-access-s2fnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:45.534406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.534391    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5" pod "9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" (UID: "9f26cfa4-d3ab-4973-bfe8-bd9173f524d5"). InnerVolumeSpecName "kube-api-access-6v5l5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:45.634142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.634085    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2fnb\" (UniqueName: \"kubernetes.io/projected/a8e30eee-5488-49a1-ad18-9b2ca9bd7224-kube-api-access-s2fnb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:45.634142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.634109    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v5l5\" (UniqueName: \"kubernetes.io/projected/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5-kube-api-access-6v5l5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:45.864273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.864218    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"]
Nov 03 14:45:45.875952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.875923    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd"]
Nov 03 14:45:45.906282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.906212    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"]
Nov 03 14:45:45.911871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:45.911851    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2"]
Nov 03 14:45:46.065849 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:46.065815    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" path="/var/lib/kubelet/pods/9f26cfa4-d3ab-4973-bfe8-bd9173f524d5/volumes"
Nov 03 14:45:46.066209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:46.066195    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" path="/var/lib/kubelet/pods/a8e30eee-5488-49a1-ad18-9b2ca9bd7224/volumes"
Nov 03 14:45:46.227353 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:46.227320    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-331c9e-b6dfd87cf-j6wq2" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.77:50052\" within 1s: context deadline exceeded"
Nov 03 14:45:46.227788 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:46.227331    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-331c9e-6d869d5455-fzvxd" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.76:50052\" within 1s: context deadline exceeded"
Nov 03 14:45:47.902462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:47.902395    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:48.106097 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:48.106065    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:48.646016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:48.645985    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:48.919662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:48.919632    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"]
Nov 03 14:45:48.920090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:48.919822    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy" containerID="cri-o://9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120" gracePeriod=30
Nov 03 14:45:49.046864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:49.046836    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:49.939856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:49.939823    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948677c5f-lc4kc_1a4df69a-9878-4445-a3ab-94d60af00096/authorino/0.log"
Nov 03 14:45:51.562248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:51.562201    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8vvn_5135bcc0-45a6-414e-910c-f68bed8be41a/limitador/0.log"
Nov 03 14:45:51.650058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:51.650021    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:45:51.650206 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:51.650035    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:45:53.959884 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:53.959851    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78fsn_5e1d2c0b-5871-481c-83b2-48f62884699d/server/0.log"
Nov 03 14:45:54.155974 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.155944    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nmtcb_07e51ae4-b9c2-4424-b835-81b5b8f33934/manager/0.log"
Nov 03 14:45:54.179513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.179486    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:45:54.312923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.312844    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.312933    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.312965    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313029    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313089 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313077    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313094    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313112    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:54.313401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313122    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313290    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313294    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:54.313401 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313333    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx\") pod \"5a781268-b0a7-4b29-afbb-eb23570b25d8\" (UID: \"5a781268-b0a7-4b29-afbb-eb23570b25d8\") "
Nov 03 14:45:54.313764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313331    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:54.313764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313662    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:45:54.313886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313811    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a781268-b0a7-4b29-afbb-eb23570b25d8-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.313886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313833    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.313886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313833    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:54.313886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313846    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.313886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.313867    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.315520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.315481    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "kube-api-access-ql7bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:54.315728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.315692    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:45:54.315801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.315780    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:45:54.315838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.315814    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo" pod "5a781268-b0a7-4b29-afbb-eb23570b25d8" (UID: "5a781268-b0a7-4b29-afbb-eb23570b25d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:45:54.385992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.385969    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kjghr_e5dafb2a-ee70-4307-8977-f1289ea765aa/proxy/0.log"
Nov 03 14:45:54.414429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.414402    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.414429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.414425    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.414429 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.414433    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.414629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.414441    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a781268-b0a7-4b29-afbb-eb23570b25d8-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.414629 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.414452    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7bx\" (UniqueName: \"kubernetes.io/projected/5a781268-b0a7-4b29-afbb-eb23570b25d8-kube-api-access-ql7bx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:45:54.567844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.567757    6481 generic.go:356] "Generic (PLEG): container finished" podID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerID="9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120" exitCode=0
Nov 03 14:45:54.567844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.567810    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"
Nov 03 14:45:54.568022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.567849    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" event={"ID":"5a781268-b0a7-4b29-afbb-eb23570b25d8","Type":"ContainerDied","Data":"9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120"}
Nov 03 14:45:54.568022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.567907    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz" event={"ID":"5a781268-b0a7-4b29-afbb-eb23570b25d8","Type":"ContainerDied","Data":"4a578de4a87c34c328bdf64994a6c7e65ccb8259e550c81cea63e8045f0d877f"}
Nov 03 14:45:54.568022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.567930    6481 scope.go:122] "RemoveContainer" containerID="9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120"
Nov 03 14:45:54.577457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.577438    6481 scope.go:122] "RemoveContainer" containerID="9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120"
Nov 03 14:45:54.577722 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:54.577705    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120\": container with ID starting with 9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120 not found: ID does not exist" containerID="9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120"
Nov 03 14:45:54.577771 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.577730    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120"} err="failed to get container status \"9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120\": rpc error: code = NotFound desc = could not find container \"9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120\": container with ID starting with 9f100e9c00e50390395715c5e710eb4113d5c7910bfb1b1fc844ade17ff4b120 not found: ID does not exist"
Nov 03 14:45:54.624102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.624079    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"]
Nov 03 14:45:54.636065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:54.636045    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-b4361f-openshift-default-756f75c897-b2bvz"]
Nov 03 14:45:56.065693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:56.065661    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" path="/var/lib/kubelet/pods/5a781268-b0a7-4b29-afbb-eb23570b25d8/volumes"
Nov 03 14:45:57.991182 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:45:57.990212    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:45:58.770150 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:58.770111    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:59.275332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:59.275300    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:45:59.491882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:45:59.491850    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:00.371661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:00.371627    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:01.649517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:01.649486    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:46:01.649937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:01.649487    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:46:01.711675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:01.711642    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:01.929637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:01.929602    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:03.063362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.063327    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"]
Nov 03 14:46:03.074806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.074777    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bfe7505-d6dpz"]
Nov 03 14:46:03.500026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.499983    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:03.606881 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.606846    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"]
Nov 03 14:46:03.607269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607256    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607270    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607285    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607291    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607300    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f90933fd-b96a-4e6e-bc8e-f2accbc0df53" containerName="payload-processing"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607306    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f90933fd-b96a-4e6e-bc8e-f2accbc0df53" containerName="payload-processing"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607314    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.607325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607319    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607331    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607336    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607350    6481 cpu_manager.go:414] "containerMap: removing container" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607355    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607368    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e5517347-66e7-402c-b301-1e29c5dcbe8b" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607373    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5517347-66e7-402c-b301-1e29c5dcbe8b" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607384    6481 cpu_manager.go:414] "containerMap: removing container" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607388    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607396    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607400    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607407    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607411    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607416    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607420    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607427    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607432    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607499    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607508    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c3947e-e956-43af-b11a-966ae8ee494f" containerName="istio-proxy"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607516    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5517347-66e7-402c-b301-1e29c5dcbe8b" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607522    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607528    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607535    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef3669f-7f28-45a7-9e72-7b705e2bc8d5" containerName="istio-proxy"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607540    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e30eee-5488-49a1-ad18-9b2ca9bd7224" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607544    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607550    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7c0fe9-d2a1-4cb5-a340-7316d9dcb065" containerName="payload-processing"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607556    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.607549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607560    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f26cfa4-d3ab-4973-bfe8-bd9173f524d5" containerName="payload-pre-processing"
Nov 03 14:46:03.608398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607565    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a781268-b0a7-4b29-afbb-eb23570b25d8" containerName="istio-proxy"
Nov 03 14:46:03.608398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607572    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f90933fd-b96a-4e6e-bc8e-f2accbc0df53" containerName="payload-processing"
Nov 03 14:46:03.608398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.607577    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.610783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.610757    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.631219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.631184    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"]
Nov 03 14:46:03.632127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.632099    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.632127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.632124    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68d0305-6712-4c63-9f73-4e6c4a5ecde5" containerName="cleanup"
Nov 03 14:46:03.632362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.632154    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.632362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.632162    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad954b63-e848-476a-a944-5140d5755c17" containerName="cleanup"
Nov 03 14:46:03.635855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.635833    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"]
Nov 03 14:46:03.635989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.635944    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.638179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.638149    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8c02610e\""
Nov 03 14:46:03.638314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.638290    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8c02610e-dockercfg-q5smq\""
Nov 03 14:46:03.650488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.650456    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"]
Nov 03 14:46:03.696509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.696483    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume\") pod \"payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") " pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.696654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.696524    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k96k\" (UniqueName: \"kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k\") pod \"payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.696654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.696601    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.696738 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.696701    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclpp\" (UniqueName: \"kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp\") pod \"payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") " pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.716568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.716531    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:03.798099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.798010    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclpp\" (UniqueName: \"kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp\") pod \"payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") " pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.798099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.798063    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume\") pod \"payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") " pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.798373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.798108    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k96k\" (UniqueName: \"kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k\") pod \"payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.798373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.798161    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.798883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.798857    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume\") pod \"payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") " pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.798998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.798946    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.820790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.820762    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k96k\" (UniqueName: \"kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k\") pod \"payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:03.820919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.820796    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclpp\" (UniqueName: \"kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp\") pod \"payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") " pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.927465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.927427    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:03.930637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.930437    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:03.950475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:03.950453    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:04.065575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.065519    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70fa97c-d6f1-444c-b8cd-8da49053182a" path="/var/lib/kubelet/pods/e70fa97c-d6f1-444c-b8cd-8da49053182a/volumes"
Nov 03 14:46:04.238416 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.238374    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"]
Nov 03 14:46:04.242676 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:46:04.242646    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ece68be_84da_4df5_a92f_bfef2c159d48.slice/crio-a365b3fe0e9d8b1797ce56fe4abd42e008ede6b58ceaa46cb84e2a020a6bfaac WatchSource:0}: Error finding container a365b3fe0e9d8b1797ce56fe4abd42e008ede6b58ceaa46cb84e2a020a6bfaac: Status 404 returned error can't find the container with id a365b3fe0e9d8b1797ce56fe4abd42e008ede6b58ceaa46cb84e2a020a6bfaac
Nov 03 14:46:04.283372 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.283351    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"]
Nov 03 14:46:04.284507 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:46:04.284482    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a8636_c4d8_4395_a4e9_f0259beb08ae.slice/crio-b31eb768ecd6d962ac6ee960ab960184520bc77b160cc762c5a552aa1cd09c58 WatchSource:0}: Error finding container b31eb768ecd6d962ac6ee960ab960184520bc77b160cc762c5a552aa1cd09c58: Status 404 returned error can't find the container with id b31eb768ecd6d962ac6ee960ab960184520bc77b160cc762c5a552aa1cd09c58
Nov 03 14:46:04.628070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.627979    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz" event={"ID":"ff4a8636-c4d8-4395-a4e9-f0259beb08ae","Type":"ContainerStarted","Data":"cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29"}
Nov 03 14:46:04.628070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.628034    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz" event={"ID":"ff4a8636-c4d8-4395-a4e9-f0259beb08ae","Type":"ContainerStarted","Data":"b31eb768ecd6d962ac6ee960ab960184520bc77b160cc762c5a552aa1cd09c58"}
Nov 03 14:46:04.628070 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.628063    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:04.629547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.629522    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z" event={"ID":"7ece68be-84da-4df5-a92f-bfef2c159d48","Type":"ContainerStarted","Data":"49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e"}
Nov 03 14:46:04.629661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.629570    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z" event={"ID":"7ece68be-84da-4df5-a92f-bfef2c159d48","Type":"ContainerStarted","Data":"a365b3fe0e9d8b1797ce56fe4abd42e008ede6b58ceaa46cb84e2a020a6bfaac"}
Nov 03 14:46:04.629661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.629591    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:04.669672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.669626    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz" podStartSLOduration=1.66961284 podStartE2EDuration="1.66961284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:04.667686066 +0000 UTC m=+2042.717717250" watchObservedRunningTime="2026-11-03 14:46:04.66961284 +0000 UTC m=+2042.719644047"
Nov 03 14:46:04.711838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:04.711788    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z" podStartSLOduration=1.711779928 podStartE2EDuration="1.711779928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:04.710682756 +0000 UTC m=+2042.760713952" watchObservedRunningTime="2026-11-03 14:46:04.711779928 +0000 UTC m=+2042.761811111"
Nov 03 14:46:05.436528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.436503    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:46:05.447812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.447791    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:46:05.522086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522056    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume\") pod \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") "
Nov 03 14:46:05.522287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522123    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n\" (UniqueName: \"kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n\") pod \"646ad667-a372-4de9-b38e-83332f77a8f5\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") "
Nov 03 14:46:05.522287 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522198    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume\") pod \"646ad667-a372-4de9-b38e-83332f77a8f5\" (UID: \"646ad667-a372-4de9-b38e-83332f77a8f5\") "
Nov 03 14:46:05.522440 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522292    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh\" (UniqueName: \"kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh\") pod \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\" (UID: \"88f5d0f1-cab0-496c-b767-68b59f7ae06c\") "
Nov 03 14:46:05.522573 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522547    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume" pod "88f5d0f1-cab0-496c-b767-68b59f7ae06c" (UID: "88f5d0f1-cab0-496c-b767-68b59f7ae06c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:46:05.522663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522590    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume" pod "646ad667-a372-4de9-b38e-83332f77a8f5" (UID: "646ad667-a372-4de9-b38e-83332f77a8f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:46:05.522825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522808    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/646ad667-a372-4de9-b38e-83332f77a8f5-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:05.522895 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.522830    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f5d0f1-cab0-496c-b767-68b59f7ae06c-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:05.524968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.524941    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n" pod "646ad667-a372-4de9-b38e-83332f77a8f5" (UID: "646ad667-a372-4de9-b38e-83332f77a8f5"). InnerVolumeSpecName "kube-api-access-rlb4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:46:05.525137 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.525111    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh" pod "88f5d0f1-cab0-496c-b767-68b59f7ae06c" (UID: "88f5d0f1-cab0-496c-b767-68b59f7ae06c"). InnerVolumeSpecName "kube-api-access-cz6mh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:46:05.623845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.623817    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz6mh\" (UniqueName: \"kubernetes.io/projected/88f5d0f1-cab0-496c-b767-68b59f7ae06c-kube-api-access-cz6mh\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:05.623845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.623839    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlb4n\" (UniqueName: \"kubernetes.io/projected/646ad667-a372-4de9-b38e-83332f77a8f5-kube-api-access-rlb4n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:05.636589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.636564    6481 generic.go:356] "Generic (PLEG): container finished" podID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerID="f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7" exitCode=0
Nov 03 14:46:05.636718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.636607    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"
Nov 03 14:46:05.636718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.636648    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" event={"ID":"88f5d0f1-cab0-496c-b767-68b59f7ae06c","Type":"ContainerDied","Data":"f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7"}
Nov 03 14:46:05.636718 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.636711    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf" event={"ID":"88f5d0f1-cab0-496c-b767-68b59f7ae06c","Type":"ContainerDied","Data":"8694b4bb049f480110eec704b2303922942cf3c8ea06e5d5adbc85788ced7748"}
Nov 03 14:46:05.636856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.636745    6481 scope.go:122] "RemoveContainer" containerID="f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7"
Nov 03 14:46:05.638255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.638207    6481 generic.go:356] "Generic (PLEG): container finished" podID="646ad667-a372-4de9-b38e-83332f77a8f5" containerID="e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3" exitCode=0
Nov 03 14:46:05.638407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.638266    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" event={"ID":"646ad667-a372-4de9-b38e-83332f77a8f5","Type":"ContainerDied","Data":"e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3"}
Nov 03 14:46:05.638407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.638287    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"
Nov 03 14:46:05.638407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.638320    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq" event={"ID":"646ad667-a372-4de9-b38e-83332f77a8f5","Type":"ContainerDied","Data":"ea8a18616294ee96b09b2c4510405c1d241678cbe2d9ada4c88c0a110c2272ee"}
Nov 03 14:46:05.647777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.647758    6481 scope.go:122] "RemoveContainer" containerID="f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7"
Nov 03 14:46:05.648094 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:46:05.648072    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7\": container with ID starting with f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7 not found: ID does not exist" containerID="f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7"
Nov 03 14:46:05.648179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.648107    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7"} err="failed to get container status \"f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7\": rpc error: code = NotFound desc = could not find container \"f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7\": container with ID starting with f5cfb60c1eca2df2db7abf9b2e955bf626b1b9e7794697ad22951890a04d4cd7 not found: ID does not exist"
Nov 03 14:46:05.648179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.648132    6481 scope.go:122] "RemoveContainer" containerID="e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3"
Nov 03 14:46:05.656803 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.656782    6481 scope.go:122] "RemoveContainer" containerID="e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3"
Nov 03 14:46:05.657059 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:46:05.657042    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3\": container with ID starting with e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3 not found: ID does not exist" containerID="e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3"
Nov 03 14:46:05.657105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.657067    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3"} err="failed to get container status \"e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3\": rpc error: code = NotFound desc = could not find container \"e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3\": container with ID starting with e9fb153eb4881a60030c5616d241b477370e50dc5d8c9f39742e3d6db58910f3 not found: ID does not exist"
Nov 03 14:46:05.688505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.688457    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"]
Nov 03 14:46:05.694933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.694903    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b4361f-59f6d9ddc9-5jffq"]
Nov 03 14:46:05.728841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.728821    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"]
Nov 03 14:46:05.735415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.735394    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b4361f-b4fdcf8df-h9vmf"]
Nov 03 14:46:05.787689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:05.787663    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:06.065815 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:06.065729    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" path="/var/lib/kubelet/pods/646ad667-a372-4de9-b38e-83332f77a8f5/volumes"
Nov 03 14:46:06.066249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:06.066215    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" path="/var/lib/kubelet/pods/88f5d0f1-cab0-496c-b767-68b59f7ae06c/volumes"
Nov 03 14:46:07.460784 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:07.460749    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:46:07.727453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:07.727378    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:07.873913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:07.873880    6481 scope.go:122] "RemoveContainer" containerID="94884579839f19dd673d9e2680f01bfc76d4c6f19915fed8e608c87fe091ff8a"
Nov 03 14:46:07.882855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:07.882832    6481 scope.go:122] "RemoveContainer" containerID="90a099c9f17a2103d9d9af4037edbd7117a4ac1ae445820dc85df72e626b99a9"
Nov 03 14:46:09.913134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:09.913101    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:11.107626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:11.107592    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:12.038459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:12.038431    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:14.096173 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:14.096142    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:15.641474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:15.641440    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:15.641839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:15.641600    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:16.043608 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:16.043572    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"]
Nov 03 14:46:16.047595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:16.047572    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0db7c9cd-n96kz"]
Nov 03 14:46:16.066522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:16.066496    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905f76c8-5ac6-43ac-b9ca-05d663fd31c6" path="/var/lib/kubelet/pods/905f76c8-5ac6-43ac-b9ca-05d663fd31c6/volumes"
Nov 03 14:46:16.304731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:16.304655    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:18.443051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:18.443017    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:20.553720 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.553684    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"]
Nov 03 14:46:20.554390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.554358    6481 cpu_manager.go:414] "containerMap: removing container" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" containerName="payload-pre-processing"
Nov 03 14:46:20.554556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.554388    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" containerName="payload-pre-processing"
Nov 03 14:46:20.554556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.554459    6481 cpu_manager.go:414] "containerMap: removing container" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerName="payload-processing"
Nov 03 14:46:20.554556 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.554467    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerName="payload-processing"
Nov 03 14:46:20.554762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.554597    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="646ad667-a372-4de9-b38e-83332f77a8f5" containerName="payload-pre-processing"
Nov 03 14:46:20.554762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.554616    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="88f5d0f1-cab0-496c-b767-68b59f7ae06c" containerName="payload-processing"
Nov 03 14:46:20.560439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.560416    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.578846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.578823    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"]
Nov 03 14:46:20.587709 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.587685    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:46:20.587846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.587747    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:46:20.587846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.587746    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:46:20.587846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.587827    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:46:20.602453 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.602430    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:20.670729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.670706    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz764\" (UniqueName: \"kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.670839 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.670763    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.670887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.670841    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.772378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.772350    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.772516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.772398    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.772516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.772483    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz764\" (UniqueName: \"kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.773011 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.772991    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.774744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.774725    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.793260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.793219    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz764\" (UniqueName: \"kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764\") pod \"maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:20.873886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:20.873823    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:21.172458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:21.172435    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"]
Nov 03 14:46:21.174762 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:46:21.174724    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc886571d_c801_41d7_a3b6_e42408f80227.slice/crio-0f0df1e5e7e114afb9573dcb8691d86815efd5db082e4d40e8c07b59e5313b2b WatchSource:0}: Error finding container 0f0df1e5e7e114afb9573dcb8691d86815efd5db082e4d40e8c07b59e5313b2b: Status 404 returned error can't find the container with id 0f0df1e5e7e114afb9573dcb8691d86815efd5db082e4d40e8c07b59e5313b2b
Nov 03 14:46:21.728528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:21.728492    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv" event={"ID":"c886571d-c801-41d7-a3b6-e42408f80227","Type":"ContainerStarted","Data":"657dcdd1c19028f3ee215dcfac0015bab30def7b9aeed857a094795468720cee"}
Nov 03 14:46:21.728970 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:21.728549    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv" event={"ID":"c886571d-c801-41d7-a3b6-e42408f80227","Type":"ContainerStarted","Data":"0f0df1e5e7e114afb9573dcb8691d86815efd5db082e4d40e8c07b59e5313b2b"}
Nov 03 14:46:21.771467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:21.771367    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv" podStartSLOduration=1.771353155 podStartE2EDuration="1.771353155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:21.769339961 +0000 UTC m=+2059.819371145" watchObservedRunningTime="2026-11-03 14:46:21.771353155 +0000 UTC m=+2059.821384337"
Nov 03 14:46:22.719054 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:22.719024    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:22.735126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:22.735098    6481 generic.go:356] "Generic (PLEG): container finished" podID="c886571d-c801-41d7-a3b6-e42408f80227" containerID="657dcdd1c19028f3ee215dcfac0015bab30def7b9aeed857a094795468720cee" exitCode=0
Nov 03 14:46:22.735592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:22.735184    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv" event={"ID":"c886571d-c801-41d7-a3b6-e42408f80227","Type":"ContainerDied","Data":"657dcdd1c19028f3ee215dcfac0015bab30def7b9aeed857a094795468720cee"}
Nov 03 14:46:23.869711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:23.869681    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:24.001550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.001472    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764\" (UniqueName: \"kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764\") pod \"c886571d-c801-41d7-a3b6-e42408f80227\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") "
Nov 03 14:46:24.001550 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.001514    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca\") pod \"c886571d-c801-41d7-a3b6-e42408f80227\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") "
Nov 03 14:46:24.001744 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.001568    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca\") pod \"c886571d-c801-41d7-a3b6-e42408f80227\" (UID: \"c886571d-c801-41d7-a3b6-e42408f80227\") "
Nov 03 14:46:24.002010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.001983    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca" pod "c886571d-c801-41d7-a3b6-e42408f80227" (UID: "c886571d-c801-41d7-a3b6-e42408f80227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:46:24.003633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.003602    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca" pod "c886571d-c801-41d7-a3b6-e42408f80227" (UID: "c886571d-c801-41d7-a3b6-e42408f80227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:46:24.003745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.003653    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764" pod "c886571d-c801-41d7-a3b6-e42408f80227" (UID: "c886571d-c801-41d7-a3b6-e42408f80227"). InnerVolumeSpecName "kube-api-access-mz764". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:46:24.103090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.103061    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz764\" (UniqueName: \"kubernetes.io/projected/c886571d-c801-41d7-a3b6-e42408f80227-kube-api-access-mz764\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:24.103090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.103089    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:24.103316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.103100    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c886571d-c801-41d7-a3b6-e42408f80227-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:24.698449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.698418    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:24.746417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.746391    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"
Nov 03 14:46:24.746569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.746395    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv" event={"ID":"c886571d-c801-41d7-a3b6-e42408f80227","Type":"ContainerDied","Data":"0f0df1e5e7e114afb9573dcb8691d86815efd5db082e4d40e8c07b59e5313b2b"}
Nov 03 14:46:24.746569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:24.746516    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0df1e5e7e114afb9573dcb8691d86815efd5db082e4d40e8c07b59e5313b2b"
Nov 03 14:46:26.696951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:26.696917    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:27.985428 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:46:27.985397    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:46:28.034940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:28.034907    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"]
Nov 03 14:46:28.079209 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:28.079180    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"]
Nov 03 14:46:28.381951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:28.381873    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:28.602943 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:28.602914    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:29.563426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:29.563384    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:29.704728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:29.704699    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"]
Nov 03 14:46:29.716468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:29.716443    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c02610e-gz4xv"]
Nov 03 14:46:29.782766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:29.782741    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:30.066082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:30.066050    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c886571d-c801-41d7-a3b6-e42408f80227" path="/var/lib/kubelet/pods/c886571d-c801-41d7-a3b6-e42408f80227/volumes"
Nov 03 14:46:31.495465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:31.495432    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:46:33.035661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.035615    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z" podUID="7ece68be-84da-4df5-a92f-bfef2c159d48" containerName="payload-processing" containerID="cri-o://49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e" gracePeriod=25
Nov 03 14:46:33.079991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.079924    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz" podUID="ff4a8636-c4d8-4395-a4e9-f0259beb08ae" containerName="payload-pre-processing" containerID="cri-o://cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29" gracePeriod=25
Nov 03 14:46:33.339048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.339027    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:33.364630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.364611    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:33.495997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.495969    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp\" (UniqueName: \"kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp\") pod \"7ece68be-84da-4df5-a92f-bfef2c159d48\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") "
Nov 03 14:46:33.496145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.496034    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume\") pod \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") "
Nov 03 14:46:33.496145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.496097    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume\") pod \"7ece68be-84da-4df5-a92f-bfef2c159d48\" (UID: \"7ece68be-84da-4df5-a92f-bfef2c159d48\") "
Nov 03 14:46:33.496306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.496174    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k\" (UniqueName: \"kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k\") pod \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\" (UID: \"ff4a8636-c4d8-4395-a4e9-f0259beb08ae\") "
Nov 03 14:46:33.496467 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.496444    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume" pod "ff4a8636-c4d8-4395-a4e9-f0259beb08ae" (UID: "ff4a8636-c4d8-4395-a4e9-f0259beb08ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:46:33.496568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.496546    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume" pod "7ece68be-84da-4df5-a92f-bfef2c159d48" (UID: "7ece68be-84da-4df5-a92f-bfef2c159d48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:46:33.497991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.497971    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp" pod "7ece68be-84da-4df5-a92f-bfef2c159d48" (UID: "7ece68be-84da-4df5-a92f-bfef2c159d48"). InnerVolumeSpecName "kube-api-access-cclpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:46:33.498079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.498048    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k" pod "ff4a8636-c4d8-4395-a4e9-f0259beb08ae" (UID: "ff4a8636-c4d8-4395-a4e9-f0259beb08ae"). InnerVolumeSpecName "kube-api-access-9k96k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:46:33.597651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.597573    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ece68be-84da-4df5-a92f-bfef2c159d48-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:33.597651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.597600    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k96k\" (UniqueName: \"kubernetes.io/projected/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-kube-api-access-9k96k\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:33.597651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.597610    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclpp\" (UniqueName: \"kubernetes.io/projected/7ece68be-84da-4df5-a92f-bfef2c159d48-kube-api-access-cclpp\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:33.597651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.597619    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff4a8636-c4d8-4395-a4e9-f0259beb08ae-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:46:33.792688 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.792653    6481 generic.go:356] "Generic (PLEG): container finished" podID="7ece68be-84da-4df5-a92f-bfef2c159d48" containerID="49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e" exitCode=0
Nov 03 14:46:33.792865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.792708    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"
Nov 03 14:46:33.792865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.792739    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z" event={"ID":"7ece68be-84da-4df5-a92f-bfef2c159d48","Type":"ContainerDied","Data":"49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e"}
Nov 03 14:46:33.792865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.792799    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z" event={"ID":"7ece68be-84da-4df5-a92f-bfef2c159d48","Type":"ContainerDied","Data":"a365b3fe0e9d8b1797ce56fe4abd42e008ede6b58ceaa46cb84e2a020a6bfaac"}
Nov 03 14:46:33.792865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.792823    6481 scope.go:122] "RemoveContainer" containerID="49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e"
Nov 03 14:46:33.794195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.794173    6481 generic.go:356] "Generic (PLEG): container finished" podID="ff4a8636-c4d8-4395-a4e9-f0259beb08ae" containerID="cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29" exitCode=0
Nov 03 14:46:33.794325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.794213    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"
Nov 03 14:46:33.794325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.794244    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz" event={"ID":"ff4a8636-c4d8-4395-a4e9-f0259beb08ae","Type":"ContainerDied","Data":"cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29"}
Nov 03 14:46:33.794325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.794280    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz" event={"ID":"ff4a8636-c4d8-4395-a4e9-f0259beb08ae","Type":"ContainerDied","Data":"b31eb768ecd6d962ac6ee960ab960184520bc77b160cc762c5a552aa1cd09c58"}
Nov 03 14:46:33.803724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.803709    6481 scope.go:122] "RemoveContainer" containerID="49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e"
Nov 03 14:46:33.803982 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:46:33.803966    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e\": container with ID starting with 49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e not found: ID does not exist" containerID="49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e"
Nov 03 14:46:33.804059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.803993    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e"} err="failed to get container status \"49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e\": rpc error: code = NotFound desc = could not find container \"49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e\": container with ID starting with 49952f0aaa7548265ea5da87fe0837fdce2fab5e8ace5ee720fab11799081e3e not found: ID does not exist"
Nov 03 14:46:33.804059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.804014    6481 scope.go:122] "RemoveContainer" containerID="cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29"
Nov 03 14:46:33.812158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.812140    6481 scope.go:122] "RemoveContainer" containerID="cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29"
Nov 03 14:46:33.812927 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:46:33.812906    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29\": container with ID starting with cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29 not found: ID does not exist" containerID="cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29"
Nov 03 14:46:33.812997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.812935    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29"} err="failed to get container status \"cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29\": rpc error: code = NotFound desc = could not find container \"cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29\": container with ID starting with cc5ec41389329e28c9733bf19dae4d2aa2f6b91e8465b1d97ba0ef2dc86f9e29 not found: ID does not exist"
Nov 03 14:46:33.832929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.832905    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:33.846765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.846743    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"]
Nov 03 14:46:33.858392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.858348    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8c02610e-6b5f48ffbf-nhf8z"]
Nov 03 14:46:33.897500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.897458    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"]
Nov 03 14:46:33.902727 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:33.902699    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c02610e-69cddb4f5-pllpz"]
Nov 03 14:46:34.064749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:34.064719    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:46:34.066193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:34.066172    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ece68be-84da-4df5-a92f-bfef2c159d48" path="/var/lib/kubelet/pods/7ece68be-84da-4df5-a92f-bfef2c159d48/volumes"
Nov 03 14:46:34.066591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:34.066577    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4a8636-c4d8-4395-a4e9-f0259beb08ae" path="/var/lib/kubelet/pods/ff4a8636-c4d8-4395-a4e9-f0259beb08ae/volumes"
Nov 03 14:46:57.984905 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:46:57.984824    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:46:58.827726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.827692    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"]
Nov 03 14:46:58.828091 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828078    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7ece68be-84da-4df5-a92f-bfef2c159d48" containerName="payload-processing"
Nov 03 14:46:58.828149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828092    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ece68be-84da-4df5-a92f-bfef2c159d48" containerName="payload-processing"
Nov 03 14:46:58.828149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828132    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c886571d-c801-41d7-a3b6-e42408f80227" containerName="revoke-keys"
Nov 03 14:46:58.828149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828137    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c886571d-c801-41d7-a3b6-e42408f80227" containerName="revoke-keys"
Nov 03 14:46:58.828313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828150    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a8636-c4d8-4395-a4e9-f0259beb08ae" containerName="payload-pre-processing"
Nov 03 14:46:58.828313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828156    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a8636-c4d8-4395-a4e9-f0259beb08ae" containerName="payload-pre-processing"
Nov 03 14:46:58.828313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828218    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ece68be-84da-4df5-a92f-bfef2c159d48" containerName="payload-processing"
Nov 03 14:46:58.828313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828244    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c886571d-c801-41d7-a3b6-e42408f80227" containerName="revoke-keys"
Nov 03 14:46:58.828313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.828252    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a8636-c4d8-4395-a4e9-f0259beb08ae" containerName="payload-pre-processing"
Nov 03 14:46:58.831419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.831399    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:58.853598 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.853574    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:46:58.853723 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.853640    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:46:58.854508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.854475    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"]
Nov 03 14:46:58.917484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.917460    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:46:58.917653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.917636    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:46:58.946297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.946276    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:58.946389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.946327    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kns7\" (UniqueName: \"kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:58.946446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:58.946409    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.047694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.047655    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kns7\" (UniqueName: \"kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.047694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.047697    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.048087 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.047780    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.048404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.048382    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.050040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.050020    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.071682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.071656    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kns7\" (UniqueName: \"kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7\") pod \"maas-api-revoke-sub-85237e45-qrsvf\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.144532 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.144505    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:46:59.447407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.447286    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"]
Nov 03 14:46:59.449747 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:46:59.449720    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd224be_badc_4e15_a2bc_c07ab287963c.slice/crio-b99220ac46f15c6bbe330a1d564b8d6c4b1fd4a4a67da4a78de818f5e5ec0799 WatchSource:0}: Error finding container b99220ac46f15c6bbe330a1d564b8d6c4b1fd4a4a67da4a78de818f5e5ec0799: Status 404 returned error can't find the container with id b99220ac46f15c6bbe330a1d564b8d6c4b1fd4a4a67da4a78de818f5e5ec0799
Nov 03 14:46:59.974526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.974491    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf" event={"ID":"7bd224be-badc-4e15-a2bc-c07ab287963c","Type":"ContainerStarted","Data":"118b4975bb5eb001cc96f9584aa0dc057c032d08e8e6cfd36f1532d4708f03ed"}
Nov 03 14:46:59.974743 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:46:59.974548    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf" event={"ID":"7bd224be-badc-4e15-a2bc-c07ab287963c","Type":"ContainerStarted","Data":"b99220ac46f15c6bbe330a1d564b8d6c4b1fd4a4a67da4a78de818f5e5ec0799"}
Nov 03 14:47:00.017838 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:00.017786    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf" podStartSLOduration=2.017774003 podStartE2EDuration="2.017774003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:00.015972633 +0000 UTC m=+2098.066003816" watchObservedRunningTime="2026-11-03 14:47:00.017774003 +0000 UTC m=+2098.067805178"
Nov 03 14:47:00.509147 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:00.509118    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:47:00.980834 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:00.980799    6481 generic.go:356] "Generic (PLEG): container finished" podID="7bd224be-badc-4e15-a2bc-c07ab287963c" containerID="118b4975bb5eb001cc96f9584aa0dc057c032d08e8e6cfd36f1532d4708f03ed" exitCode=0
Nov 03 14:47:00.981022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:00.980882    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf" event={"ID":"7bd224be-badc-4e15-a2bc-c07ab287963c","Type":"ContainerDied","Data":"118b4975bb5eb001cc96f9584aa0dc057c032d08e8e6cfd36f1532d4708f03ed"}
Nov 03 14:47:02.118757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.118729    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:47:02.181052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.181023    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7\" (UniqueName: \"kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7\") pod \"7bd224be-badc-4e15-a2bc-c07ab287963c\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") "
Nov 03 14:47:02.181240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.181139    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca\") pod \"7bd224be-badc-4e15-a2bc-c07ab287963c\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") "
Nov 03 14:47:02.181240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.181172    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca\") pod \"7bd224be-badc-4e15-a2bc-c07ab287963c\" (UID: \"7bd224be-badc-4e15-a2bc-c07ab287963c\") "
Nov 03 14:47:02.181601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.181555    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca" pod "7bd224be-badc-4e15-a2bc-c07ab287963c" (UID: "7bd224be-badc-4e15-a2bc-c07ab287963c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:47:02.183531 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.183504    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7" pod "7bd224be-badc-4e15-a2bc-c07ab287963c" (UID: "7bd224be-badc-4e15-a2bc-c07ab287963c"). InnerVolumeSpecName "kube-api-access-9kns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:47:02.184088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.184067    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca" pod "7bd224be-badc-4e15-a2bc-c07ab287963c" (UID: "7bd224be-badc-4e15-a2bc-c07ab287963c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:47:02.282865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.282807    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:47:02.282865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.282832    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd224be-badc-4e15-a2bc-c07ab287963c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:47:02.282865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.282844    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kns7\" (UniqueName: \"kubernetes.io/projected/7bd224be-badc-4e15-a2bc-c07ab287963c-kube-api-access-9kns7\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:47:02.992265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.992214    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"
Nov 03 14:47:02.992444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.992260    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf" event={"ID":"7bd224be-badc-4e15-a2bc-c07ab287963c","Type":"ContainerDied","Data":"b99220ac46f15c6bbe330a1d564b8d6c4b1fd4a4a67da4a78de818f5e5ec0799"}
Nov 03 14:47:02.992444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:02.992337    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b99220ac46f15c6bbe330a1d564b8d6c4b1fd4a4a67da4a78de818f5e5ec0799"
Nov 03 14:47:07.956210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:07.956179    6481 scope.go:122] "RemoveContainer" containerID="c7c46201198dfc12aee4d12d6eee1ca2190dc878808957dc7c4720610f9ce9c3"
Nov 03 14:47:08.336905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.336825    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"]
Nov 03 14:47:08.337448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.337420    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7bd224be-badc-4e15-a2bc-c07ab287963c" containerName="revoke-keys"
Nov 03 14:47:08.337448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.337440    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd224be-badc-4e15-a2bc-c07ab287963c" containerName="revoke-keys"
Nov 03 14:47:08.337612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.337519    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd224be-badc-4e15-a2bc-c07ab287963c" containerName="revoke-keys"
Nov 03 14:47:08.342666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.342640    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.364706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.364680    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"]
Nov 03 14:47:08.440773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.440738    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.440960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.440812    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.440960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.440850    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.440960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.440867    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.440960 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.440928    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.441145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.441005    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.441145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.441038    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.441145 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.441067    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpxs\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.441333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.441175    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.541822 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.541792    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.541832    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpxs\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.541854    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542039    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542155    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542190    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542218    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542277    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542319    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542425    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542646 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542623    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542629    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542711 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542677    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.542879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.542731    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.544281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.544262    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.544386 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.544360    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.562757 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.562738    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpxs\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.562928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.562906    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-mcrdq\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.657396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.657376    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:08.973666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.973585    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"]
Nov 03 14:47:08.976730 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:47:08.976697    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26794db_ebe2_44b0_aca5_5b2bf5374762.slice/crio-d8097ef083c06fbf858e04f4428c03d30cb3aa2a6344df57c4f434d72bda1d2e WatchSource:0}: Error finding container d8097ef083c06fbf858e04f4428c03d30cb3aa2a6344df57c4f434d72bda1d2e: Status 404 returned error can't find the container with id d8097ef083c06fbf858e04f4428c03d30cb3aa2a6344df57c4f434d72bda1d2e
Nov 03 14:47:08.979100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.979066    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:47:08.979164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.979132    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:47:08.979164 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:08.979159    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:47:09.026665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:09.026625    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" event={"ID":"f26794db-ebe2-44b0-aca5-5b2bf5374762","Type":"ContainerStarted","Data":"d8097ef083c06fbf858e04f4428c03d30cb3aa2a6344df57c4f434d72bda1d2e"}
Nov 03 14:47:10.033405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:10.033365    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" event={"ID":"f26794db-ebe2-44b0-aca5-5b2bf5374762","Type":"ContainerStarted","Data":"74ee0abadbdf9d9992906b7f247e106ef9dce51829434178950dc52dd4bb26e6"}
Nov 03 14:47:10.090871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:10.090822    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podStartSLOduration=2.090807595 podStartE2EDuration="2.090807595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:10.089858385 +0000 UTC m=+2108.139889569" watchObservedRunningTime="2026-11-03 14:47:10.090807595 +0000 UTC m=+2108.140838778"
Nov 03 14:47:10.658186 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:10.658149    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:10.659678 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:10.659656    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:10.659734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:10.659701    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:10.820191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:10.820153    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:47:11.658258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:11.658207    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:11.658666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:11.658291    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:12.657903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:12.657855    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:12.658117 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:12.657927    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:12.698774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:12.698739    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:47:13.658197 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:13.658161    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:13.658393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:13.658218    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:14.658690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:14.658652    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:14.659088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:14.658707    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:15.658637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:15.658604    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:15.658827 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:15.658659    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:16.657928 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:16.657893    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Nov 03 14:47:16.658093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:16.657959    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Nov 03 14:47:18.129015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:18.128983    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:18.129963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:18.129932    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:18.130674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:18.130654    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 14:47:27.985848 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:47:27.985809    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:47:33.975451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:33.975414    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:47:39.553013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.552981    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"]
Nov 03 14:47:39.556879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.556856    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.572927 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.572905    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:47:39.573048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.573023    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:47:39.584925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.584864    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:47:39.584925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.584917    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:47:39.588154 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.588128    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"]
Nov 03 14:47:39.657208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.657176    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.657435 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.657264    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.657491 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.657426    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2rj\" (UniqueName: \"kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.758476 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.758448    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2rj\" (UniqueName: \"kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.758614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.758495    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.758614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.758538    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.759142 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.759118    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.760832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.760805    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.782995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.782968    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2rj\" (UniqueName: \"kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj\") pod \"maas-api-revoke-sub-f904bff9-8qgn5\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:39.872270 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:39.872178    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:40.178183 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:40.178156    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"]
Nov 03 14:47:40.179716 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:47:40.179687    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2f4c5c7_650d_4e58_84ee_cf6e990fcec9.slice/crio-a72054e765d2f5124adb7f09a526ade5c87005bd760a2fde563b097c798b917b WatchSource:0}: Error finding container a72054e765d2f5124adb7f09a526ade5c87005bd760a2fde563b097c798b917b: Status 404 returned error can't find the container with id a72054e765d2f5124adb7f09a526ade5c87005bd760a2fde563b097c798b917b
Nov 03 14:47:40.335056 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:40.335025    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5" event={"ID":"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9","Type":"ContainerStarted","Data":"a72054e765d2f5124adb7f09a526ade5c87005bd760a2fde563b097c798b917b"}
Nov 03 14:47:41.341296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:41.341201    6481 generic.go:356] "Generic (PLEG): container finished" podID="a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" containerID="f2f1e6ececad53505a39a70bc0ecd1d3a4b577f08e2803091cea4c62ebd5824e" exitCode=0
Nov 03 14:47:41.341659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:41.341291    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5" event={"ID":"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9","Type":"ContainerDied","Data":"f2f1e6ececad53505a39a70bc0ecd1d3a4b577f08e2803091cea4c62ebd5824e"}
Nov 03 14:47:42.476794 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.476770    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:42.589762 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.589734    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj\" (UniqueName: \"kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj\") pod \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") "
Nov 03 14:47:42.589930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.589789    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca\") pod \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") "
Nov 03 14:47:42.589930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.589856    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca\") pod \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\" (UID: \"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9\") "
Nov 03 14:47:42.590158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.590136    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca" pod "a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" (UID: "a2f4c5c7-650d-4e58-84ee-cf6e990fcec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:47:42.590320 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.590305    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:47:42.591891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.591869    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca" pod "a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" (UID: "a2f4c5c7-650d-4e58-84ee-cf6e990fcec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:47:42.591979 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.591946    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj" pod "a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" (UID: "a2f4c5c7-650d-4e58-84ee-cf6e990fcec9"). InnerVolumeSpecName "kube-api-access-vs2rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:47:42.691502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.691477    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs2rj\" (UniqueName: \"kubernetes.io/projected/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-kube-api-access-vs2rj\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:47:42.691502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:42.691501    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:47:43.352860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:43.352831    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"
Nov 03 14:47:43.353048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:43.352861    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5" event={"ID":"a2f4c5c7-650d-4e58-84ee-cf6e990fcec9","Type":"ContainerDied","Data":"a72054e765d2f5124adb7f09a526ade5c87005bd760a2fde563b097c798b917b"}
Nov 03 14:47:43.353048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:43.352911    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a72054e765d2f5124adb7f09a526ade5c87005bd760a2fde563b097c798b917b"
Nov 03 14:47:50.955581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:50.955543    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:47:56.985151 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:47:56.985112    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:47:58.297645 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:47:58.297570    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:48:00.637338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:00.637306    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:48:01.873359 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:01.873322    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:48:07.988887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:07.988850    6481 scope.go:122] "RemoveContainer" containerID="76000b377f60950e320dda180b8cc4f5b110401988a77cfcd5ef2f836c3df43f"
Nov 03 14:48:12.612169 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.612137    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"]
Nov 03 14:48:12.612616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.612600    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" containerName="revoke-keys"
Nov 03 14:48:12.612670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.612617    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" containerName="revoke-keys"
Nov 03 14:48:12.612715 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.612693    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" containerName="revoke-keys"
Nov 03 14:48:12.617105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.617086    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.637634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.637600    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"]
Nov 03 14:48:12.646536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.646509    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:48:12.646536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.646526    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:48:12.646760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.646523    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:48:12.646760 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.646592    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:48:12.649101 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.649074    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.649214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.649129    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.649284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.649250    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpgt9\" (UniqueName: \"kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.651587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.651567    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:48:12.750622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.750590    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpgt9\" (UniqueName: \"kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.750780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.750697    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.750780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.750729    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.751492 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.751466    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.752973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.752952    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.771024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.771002    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpgt9\" (UniqueName: \"kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9\") pod \"maas-api-revoke-sub-f03629db-k5frq\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:12.881129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.881054    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:48:12.932027 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:12.931997    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:13.231602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:13.231573    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"]
Nov 03 14:48:13.233305 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:48:13.233278    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61557c54_a689_4e12_8c38_91db501f915c.slice/crio-bc16a1b7b832cb116d2cc32e7753d11436d9d06059950fb200c473ccc32d490c WatchSource:0}: Error finding container bc16a1b7b832cb116d2cc32e7753d11436d9d06059950fb200c473ccc32d490c: Status 404 returned error can't find the container with id bc16a1b7b832cb116d2cc32e7753d11436d9d06059950fb200c473ccc32d490c
Nov 03 14:48:13.646006 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:13.645971    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq" event={"ID":"61557c54-a689-4e12-8c38-91db501f915c","Type":"ContainerStarted","Data":"40d213990e9819475b5edac4290b60edce1402c800ef9ee702bca5d5d62f6232"}
Nov 03 14:48:13.646506 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:13.646028    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq" event={"ID":"61557c54-a689-4e12-8c38-91db501f915c","Type":"ContainerStarted","Data":"bc16a1b7b832cb116d2cc32e7753d11436d9d06059950fb200c473ccc32d490c"}
Nov 03 14:48:13.686791 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:13.686744    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq" podStartSLOduration=1.686731266 podStartE2EDuration="1.686731266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:48:13.686162237 +0000 UTC m=+2171.736193420" watchObservedRunningTime="2026-11-03 14:48:13.686731266 +0000 UTC m=+2171.736762448"
Nov 03 14:48:14.652922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:14.652889    6481 generic.go:356] "Generic (PLEG): container finished" podID="61557c54-a689-4e12-8c38-91db501f915c" containerID="40d213990e9819475b5edac4290b60edce1402c800ef9ee702bca5d5d62f6232" exitCode=0
Nov 03 14:48:14.653376 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:14.652988    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq" event={"ID":"61557c54-a689-4e12-8c38-91db501f915c","Type":"ContainerDied","Data":"40d213990e9819475b5edac4290b60edce1402c800ef9ee702bca5d5d62f6232"}
Nov 03 14:48:15.795181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.795155    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:15.879460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.879432    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca\") pod \"61557c54-a689-4e12-8c38-91db501f915c\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") "
Nov 03 14:48:15.879602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.879526    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca\") pod \"61557c54-a689-4e12-8c38-91db501f915c\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") "
Nov 03 14:48:15.879602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.879564    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9\" (UniqueName: \"kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9\") pod \"61557c54-a689-4e12-8c38-91db501f915c\" (UID: \"61557c54-a689-4e12-8c38-91db501f915c\") "
Nov 03 14:48:15.879869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.879845    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca" pod "61557c54-a689-4e12-8c38-91db501f915c" (UID: "61557c54-a689-4e12-8c38-91db501f915c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:48:15.881545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.881521    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9" pod "61557c54-a689-4e12-8c38-91db501f915c" (UID: "61557c54-a689-4e12-8c38-91db501f915c"). InnerVolumeSpecName "kube-api-access-qpgt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:48:15.881652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.881576    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca" pod "61557c54-a689-4e12-8c38-91db501f915c" (UID: "61557c54-a689-4e12-8c38-91db501f915c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:48:15.981220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.981160    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61557c54-a689-4e12-8c38-91db501f915c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:15.981220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.981183    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61557c54-a689-4e12-8c38-91db501f915c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:15.981220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:15.981194    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpgt9\" (UniqueName: \"kubernetes.io/projected/61557c54-a689-4e12-8c38-91db501f915c-kube-api-access-qpgt9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:16.665403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:16.665374    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"
Nov 03 14:48:16.665577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:16.665373    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq" event={"ID":"61557c54-a689-4e12-8c38-91db501f915c","Type":"ContainerDied","Data":"bc16a1b7b832cb116d2cc32e7753d11436d9d06059950fb200c473ccc32d490c"}
Nov 03 14:48:16.665577 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:16.665502    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc16a1b7b832cb116d2cc32e7753d11436d9d06059950fb200c473ccc32d490c"
Nov 03 14:48:24.082778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:24.082743    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:48:28.012113 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:48:28.012067    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:48:33.198778 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.198743    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"]
Nov 03 14:48:33.199312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.199176    6481 cpu_manager.go:414] "containerMap: removing container" podUID="61557c54-a689-4e12-8c38-91db501f915c" containerName="revoke-keys"
Nov 03 14:48:33.199312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.199192    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61557c54-a689-4e12-8c38-91db501f915c" containerName="revoke-keys"
Nov 03 14:48:33.199312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.199300    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="61557c54-a689-4e12-8c38-91db501f915c" containerName="revoke-keys"
Nov 03 14:48:33.202465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.202444    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.221002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.220968    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:48:33.221002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.220968    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:48:33.230282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.230257    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:48:33.230508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.230466    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:48:33.234000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.233976    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"]
Nov 03 14:48:33.323473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.323447    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.323624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.323487    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.323624 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.323617    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bgv\" (UniqueName: \"kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.424833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.424796    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.424961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.424845    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.424961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.424914    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bgv\" (UniqueName: \"kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.425575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.425554    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.427360 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.427335    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.445627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.445602    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bgv\" (UniqueName: \"kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv\") pod \"maas-api-revoke-sub-0165d7d2-v4bp8\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.519517 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.519461    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:33.823062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:33.823038    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"]
Nov 03 14:48:33.824868 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:48:33.824837    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9105326e_6185_47f7_b4e0_dd4f1f61f9cf.slice/crio-eae2d342cda02317c7e733a2d144267ceabb244f75e67c95377a0911bcb6f887 WatchSource:0}: Error finding container eae2d342cda02317c7e733a2d144267ceabb244f75e67c95377a0911bcb6f887: Status 404 returned error can't find the container with id eae2d342cda02317c7e733a2d144267ceabb244f75e67c95377a0911bcb6f887
Nov 03 14:48:34.818872 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:34.818838    6481 generic.go:356] "Generic (PLEG): container finished" podID="9105326e-6185-47f7-b4e0-dd4f1f61f9cf" containerID="7355b2162ede521290fd85c8f270fc70888ee68a079482d0496f7a712e9cf9a7" exitCode=0
Nov 03 14:48:34.819290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:34.818925    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8" event={"ID":"9105326e-6185-47f7-b4e0-dd4f1f61f9cf","Type":"ContainerDied","Data":"7355b2162ede521290fd85c8f270fc70888ee68a079482d0496f7a712e9cf9a7"}
Nov 03 14:48:34.819290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:34.818974    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8" event={"ID":"9105326e-6185-47f7-b4e0-dd4f1f61f9cf","Type":"ContainerStarted","Data":"eae2d342cda02317c7e733a2d144267ceabb244f75e67c95377a0911bcb6f887"}
Nov 03 14:48:35.962707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:35.962671    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:36.048866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.048834    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv\" (UniqueName: \"kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv\") pod \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") "
Nov 03 14:48:36.049022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.048876    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca\") pod \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") "
Nov 03 14:48:36.049088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.049057    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca\") pod \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\" (UID: \"9105326e-6185-47f7-b4e0-dd4f1f61f9cf\") "
Nov 03 14:48:36.049409 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.049376    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca" pod "9105326e-6185-47f7-b4e0-dd4f1f61f9cf" (UID: "9105326e-6185-47f7-b4e0-dd4f1f61f9cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:48:36.049539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.049444    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:36.051505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.051477    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca" pod "9105326e-6185-47f7-b4e0-dd4f1f61f9cf" (UID: "9105326e-6185-47f7-b4e0-dd4f1f61f9cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:48:36.051505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.051488    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv" pod "9105326e-6185-47f7-b4e0-dd4f1f61f9cf" (UID: "9105326e-6185-47f7-b4e0-dd4f1f61f9cf"). InnerVolumeSpecName "kube-api-access-q2bgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:48:36.149998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.149938    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bgv\" (UniqueName: \"kubernetes.io/projected/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-kube-api-access-q2bgv\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:36.149998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.149966    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9105326e-6185-47f7-b4e0-dd4f1f61f9cf-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:36.830421 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.830389    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"
Nov 03 14:48:36.830599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.830391    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8" event={"ID":"9105326e-6185-47f7-b4e0-dd4f1f61f9cf","Type":"ContainerDied","Data":"eae2d342cda02317c7e733a2d144267ceabb244f75e67c95377a0911bcb6f887"}
Nov 03 14:48:36.830599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:36.830517    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae2d342cda02317c7e733a2d144267ceabb244f75e67c95377a0911bcb6f887"
Nov 03 14:48:44.307637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.307606    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"]
Nov 03 14:48:44.308085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.308011    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9105326e-6185-47f7-b4e0-dd4f1f61f9cf" containerName="revoke-keys"
Nov 03 14:48:44.308085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.308023    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9105326e-6185-47f7-b4e0-dd4f1f61f9cf" containerName="revoke-keys"
Nov 03 14:48:44.308172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.308096    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9105326e-6185-47f7-b4e0-dd4f1f61f9cf" containerName="revoke-keys"
Nov 03 14:48:44.311324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.311304    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.332061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.332037    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"]
Nov 03 14:48:44.340836 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.340814    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:48:44.340968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.340843    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:48:44.340968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.340863    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:48:44.340968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.340887    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:48:44.424176 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.424139    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.424380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.424291    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.424380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.424325    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxn4\" (UniqueName: \"kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.525093 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.525061    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.525292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.525116    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.525292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.525249    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxn4\" (UniqueName: \"kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.525754 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.525728    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.527431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.527410    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.546220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.546198    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxn4\" (UniqueName: \"kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4\") pod \"maas-api-revoke-sub-6b19338f-hb4hw\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.624220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.624163    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:44.925781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:44.925754    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"]
Nov 03 14:48:44.926870 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:48:44.926843    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf61615_a11d_4683_8494_8bbd50b00d7e.slice/crio-abe9f34036e2b131476eb0978d35ef2d949ea684077858901e87bcfdbd8fff17 WatchSource:0}: Error finding container abe9f34036e2b131476eb0978d35ef2d949ea684077858901e87bcfdbd8fff17: Status 404 returned error can't find the container with id abe9f34036e2b131476eb0978d35ef2d949ea684077858901e87bcfdbd8fff17
Nov 03 14:48:45.877415 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:45.877376    6481 generic.go:356] "Generic (PLEG): container finished" podID="3bf61615-a11d-4683-8494-8bbd50b00d7e" containerID="5ca27c54f1430bdfb9dd9ed7588871f0cca36ada323cfdd3b6576e77e96e185b" exitCode=0
Nov 03 14:48:45.877813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:45.877460    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw" event={"ID":"3bf61615-a11d-4683-8494-8bbd50b00d7e","Type":"ContainerDied","Data":"5ca27c54f1430bdfb9dd9ed7588871f0cca36ada323cfdd3b6576e77e96e185b"}
Nov 03 14:48:45.877813 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:45.877516    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw" event={"ID":"3bf61615-a11d-4683-8494-8bbd50b00d7e","Type":"ContainerStarted","Data":"abe9f34036e2b131476eb0978d35ef2d949ea684077858901e87bcfdbd8fff17"}
Nov 03 14:48:47.016767 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.016741    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:47.149033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.148951    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca\") pod \"3bf61615-a11d-4683-8494-8bbd50b00d7e\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") "
Nov 03 14:48:47.149198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.149114    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca\") pod \"3bf61615-a11d-4683-8494-8bbd50b00d7e\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") "
Nov 03 14:48:47.149198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.149161    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4\" (UniqueName: \"kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4\") pod \"3bf61615-a11d-4683-8494-8bbd50b00d7e\" (UID: \"3bf61615-a11d-4683-8494-8bbd50b00d7e\") "
Nov 03 14:48:47.149525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.149499    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca" pod "3bf61615-a11d-4683-8494-8bbd50b00d7e" (UID: "3bf61615-a11d-4683-8494-8bbd50b00d7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:48:47.151122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.151100    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca" pod "3bf61615-a11d-4683-8494-8bbd50b00d7e" (UID: "3bf61615-a11d-4683-8494-8bbd50b00d7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:48:47.151185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.151158    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4" pod "3bf61615-a11d-4683-8494-8bbd50b00d7e" (UID: "3bf61615-a11d-4683-8494-8bbd50b00d7e"). InnerVolumeSpecName "kube-api-access-lhxn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:48:47.250247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.250147    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:47.250247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.250178    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxn4\" (UniqueName: \"kubernetes.io/projected/3bf61615-a11d-4683-8494-8bbd50b00d7e-kube-api-access-lhxn4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:47.250247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.250195    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bf61615-a11d-4683-8494-8bbd50b00d7e-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:48:47.890312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.890279    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"
Nov 03 14:48:47.890312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.890293    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw" event={"ID":"3bf61615-a11d-4683-8494-8bbd50b00d7e","Type":"ContainerDied","Data":"abe9f34036e2b131476eb0978d35ef2d949ea684077858901e87bcfdbd8fff17"}
Nov 03 14:48:47.890582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:47.890343    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abe9f34036e2b131476eb0978d35ef2d949ea684077858901e87bcfdbd8fff17"
Nov 03 14:48:55.757305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:55.757268    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:48:56.048946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:56.048874    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"]
Nov 03 14:48:56.065538 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:56.065508    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cf995c2-nprpq"]
Nov 03 14:48:57.256253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.255945    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"]
Nov 03 14:48:57.299265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.295708    6481 cpu_manager.go:414] "containerMap: removing container" podUID="3bf61615-a11d-4683-8494-8bbd50b00d7e" containerName="revoke-keys"
Nov 03 14:48:57.299265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.295741    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf61615-a11d-4683-8494-8bbd50b00d7e" containerName="revoke-keys"
Nov 03 14:48:57.299265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.295866    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf61615-a11d-4683-8494-8bbd50b00d7e" containerName="revoke-keys"
Nov 03 14:48:57.341393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.338404    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"]
Nov 03 14:48:57.341393 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.338521    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.409252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.396520    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-ea8adfc6-kserve-self-signed-certs\""
Nov 03 14:48:57.438662 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.438479    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:48:57.589474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.576411    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sksrx\" (UniqueName: \"kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.589856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.589819    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.590085 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.590062    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.590277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.590258    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.617251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.613635    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.617251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.613717    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.715855 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.715811    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.731747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.731716    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.732479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.732456    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.733160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.733138    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.733368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.733347    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.733625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.733606    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sksrx\" (UniqueName: \"kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.734301 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.716535    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.734806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.734787    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.735311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.735291    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.825252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.813881    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:57.831982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.831952    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"]
Nov 03 14:48:58.014254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.832400    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:58.014254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:57.849942    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sksrx\" (UniqueName: \"kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx\") pod \"e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:58.029772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.026790    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:48:58.033260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.032058    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"]
Nov 03 14:48:58.033260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.032177    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.098249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.097988    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-ea8adfc6-epp-sa-dockercfg-wjcz2\""
Nov 03 14:48:58.106251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.099516    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb99f\" (UniqueName: \"kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f\") pod \"e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.106251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.099571    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs\") pod \"e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.176254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.160917    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a35533f-d107-42ec-b252-df501d6e69e5" path="/var/lib/kubelet/pods/2a35533f-d107-42ec-b252-df501d6e69e5/volumes"
Nov 03 14:48:58.176519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.164449    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:48:58.211589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.211553    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb99f\" (UniqueName: \"kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f\") pod \"e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.211807 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.211789    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs\") pod \"e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.238263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.231856    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs\") pod \"e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.307256 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.298908    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb99f\" (UniqueName: \"kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f\") pod \"e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") " pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.428710 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:58.428428    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:48:58.524580 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:48:58.524242    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:48:59.158576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:59.150977    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"]
Nov 03 14:48:59.220904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:59.220872    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"]
Nov 03 14:48:59.221701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:59.221668    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d" event={"ID":"4ecf21f3-c878-405d-9684-86c2008d43d1","Type":"ContainerStarted","Data":"95b595e2583346336824a03bf8a63523a21de7f026957f99a058848b02d1f451"}
Nov 03 14:48:59.222129 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:48:59.222107    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8f6cf1_f473_4e21_8027_19b20f8a868d.slice/crio-4f65de205abfd3d00a4d5768ffba85a8888bb255df38a55269dcea678166ed33 WatchSource:0}: Error finding container 4f65de205abfd3d00a4d5768ffba85a8888bb255df38a55269dcea678166ed33: Status 404 returned error can't find the container with id 4f65de205abfd3d00a4d5768ffba85a8888bb255df38a55269dcea678166ed33
Nov 03 14:48:59.225997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:59.224891    6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:48:59.861639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:48:59.861602    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:49:00.230327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:00.230279    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" event={"ID":"0e8f6cf1-f473-4e21-8027-19b20f8a868d","Type":"ContainerStarted","Data":"4f65de205abfd3d00a4d5768ffba85a8888bb255df38a55269dcea678166ed33"}
Nov 03 14:49:00.232240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:00.232190    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d" event={"ID":"4ecf21f3-c878-405d-9684-86c2008d43d1","Type":"ContainerStarted","Data":"a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3"}
Nov 03 14:49:00.278837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:00.278781    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d" podStartSLOduration=3.278763337 podStartE2EDuration="3.278763337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:49:00.277786481 +0000 UTC m=+2218.327817665" watchObservedRunningTime="2026-11-03 14:49:00.278763337 +0000 UTC m=+2218.328794523"
Nov 03 14:49:00.739432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:00.739398    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:49:01.165208 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:01.165177    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:49:01.240578 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:01.240543    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" event={"ID":"0e8f6cf1-f473-4e21-8027-19b20f8a868d","Type":"ContainerStarted","Data":"a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9"}
Nov 03 14:49:01.240766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:01.240749    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:49:01.284544 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:01.284501    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podStartSLOduration=2.622802105 podStartE2EDuration="4.28448998s" totalImagesPullingTime="1.661687875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:49:01.282750547 +0000 UTC m=+2219.332781730" watchObservedRunningTime="2026-11-03 14:49:01.28448998 +0000 UTC m=+2219.334521163"
Nov 03 14:49:01.590969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:01.590889    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:49:07.040381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:07.040345    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"]
Nov 03 14:49:07.051669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:07.051641    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9140577d-km2wp"]
Nov 03 14:49:08.066175 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:08.066143    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3e51ca-e926-42ba-bb51-4c5e129bc314" path="/var/lib/kubelet/pods/bb3e51ca-e926-42ba-bb51-4c5e129bc314/volumes"
Nov 03 14:49:08.165580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:08.165547    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:49:08.177620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:08.177598    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:49:08.342591 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:08.342519    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:49:19.908412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:19.908376    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:49:20.134041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:20.134006    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:49:23.073442 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:23.073402    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"]
Nov 03 14:49:23.084348 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:23.084319    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459bfecc-fp7pv"]
Nov 03 14:49:24.066155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:24.066120    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39953a3d-b932-4503-91e7-90f91d010a31" path="/var/lib/kubelet/pods/39953a3d-b932-4503-91e7-90f91d010a31/volumes"
Nov 03 14:49:27.985249 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:49:27.985193    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:49:32.252656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:32.252626    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:49:34.038962 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:34.038929    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"]
Nov 03 14:49:34.050495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:34.050471    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7399150-r4qvh"]
Nov 03 14:49:34.065907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:34.065864    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63233c16-25af-4833-bbd1-8f22a37075b1" path="/var/lib/kubelet/pods/63233c16-25af-4833-bbd1-8f22a37075b1/volumes"
Nov 03 14:49:40.981509 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:40.981467    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:49:46.039672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:46.039583    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"]
Nov 03 14:49:46.050792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:46.050759    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fecba05-gbv5f"]
Nov 03 14:49:46.066185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:46.066155    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cfc825-e1b1-46e6-b6d0-65925d95b386" path="/var/lib/kubelet/pods/b6cfc825-e1b1-46e6-b6d0-65925d95b386/volumes"
Nov 03 14:49:49.623414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:49.623383    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:49:57.042941 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:57.042902    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"]
Nov 03 14:49:57.054010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:57.053977    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ad6b4f5-zqrbz"]
Nov 03 14:49:57.985507 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:49:57.985468    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:49:58.065706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:49:58.065665    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af9f81e-594b-402c-9094-7756765523b0" path="/var/lib/kubelet/pods/0af9f81e-594b-402c-9094-7756765523b0/volumes"
Nov 03 14:50:00.847494 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:00.847467    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:01.598921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:01.598883    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:50:01.959739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:01.959708    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"]
Nov 03 14:50:01.971323 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:01.971292    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"]
Nov 03 14:50:02.087385 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:02.087356    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:02.253050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:02.252959    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:50:02.312473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:02.312444    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:08.034284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:08.034255    6481 scope.go:122] "RemoveContainer" containerID="b2189d98e938007dd4001480331b3017ab7fb8bca2ca80e2aae40df51926a262"
Nov 03 14:50:08.152405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:08.152363    6481 scope.go:122] "RemoveContainer" containerID="6d0dde4df26e708c8e7604a9be0090db20de41336e17e761f98e21829430aecd"
Nov 03 14:50:09.213007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:09.212964    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:50:12.253100 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:12.253064    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:50:16.981575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:16.981520    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main" containerID="cri-o://a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9" gracePeriod=45
Nov 03 14:50:16.991020 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:16.990966    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d" podUID="4ecf21f3-c878-405d-9684-86c2008d43d1" containerName="main" containerID="cri-o://a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3" gracePeriod=45
Nov 03 14:50:17.251368 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.251298    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:50:17.303520 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303492    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache\") pod \"4ecf21f3-c878-405d-9684-86c2008d43d1\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") "
Nov 03 14:50:17.303669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303566    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx\" (UniqueName: \"kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx\") pod \"4ecf21f3-c878-405d-9684-86c2008d43d1\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") "
Nov 03 14:50:17.303669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303612    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir\") pod \"4ecf21f3-c878-405d-9684-86c2008d43d1\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") "
Nov 03 14:50:17.303669 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303646    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home\") pod \"4ecf21f3-c878-405d-9684-86c2008d43d1\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") "
Nov 03 14:50:17.303817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303697    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs\") pod \"4ecf21f3-c878-405d-9684-86c2008d43d1\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") "
Nov 03 14:50:17.303817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303729    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm\") pod \"4ecf21f3-c878-405d-9684-86c2008d43d1\" (UID: \"4ecf21f3-c878-405d-9684-86c2008d43d1\") "
Nov 03 14:50:17.303936 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303801    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache" pod "4ecf21f3-c878-405d-9684-86c2008d43d1" (UID: "4ecf21f3-c878-405d-9684-86c2008d43d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:50:17.303997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303912    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir" pod "4ecf21f3-c878-405d-9684-86c2008d43d1" (UID: "4ecf21f3-c878-405d-9684-86c2008d43d1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:50:17.303997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.303979    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home" pod "4ecf21f3-c878-405d-9684-86c2008d43d1" (UID: "4ecf21f3-c878-405d-9684-86c2008d43d1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:50:17.304146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.304129    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:17.304201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.304149    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:17.304201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.304165    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:17.305911 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.305882    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs" pod "4ecf21f3-c878-405d-9684-86c2008d43d1" (UID: "4ecf21f3-c878-405d-9684-86c2008d43d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:50:17.306130 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.306109    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm" pod "4ecf21f3-c878-405d-9684-86c2008d43d1" (UID: "4ecf21f3-c878-405d-9684-86c2008d43d1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:50:17.306218 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.306133    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx" pod "4ecf21f3-c878-405d-9684-86c2008d43d1" (UID: "4ecf21f3-c878-405d-9684-86c2008d43d1"). InnerVolumeSpecName "kube-api-access-sksrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:50:17.405697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.405664    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sksrx\" (UniqueName: \"kubernetes.io/projected/4ecf21f3-c878-405d-9684-86c2008d43d1-kube-api-access-sksrx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:17.405697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.405692    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ecf21f3-c878-405d-9684-86c2008d43d1-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:17.405697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.405703    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ecf21f3-c878-405d-9684-86c2008d43d1-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:17.807447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.807406    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:17.926146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.926116    6481 generic.go:356] "Generic (PLEG): container finished" podID="4ecf21f3-c878-405d-9684-86c2008d43d1" containerID="a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3" exitCode=0
Nov 03 14:50:17.926346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.926163    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"
Nov 03 14:50:17.926346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.926205    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d" event={"ID":"4ecf21f3-c878-405d-9684-86c2008d43d1","Type":"ContainerDied","Data":"a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3"}
Nov 03 14:50:17.926346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.926274    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d" event={"ID":"4ecf21f3-c878-405d-9684-86c2008d43d1","Type":"ContainerDied","Data":"95b595e2583346336824a03bf8a63523a21de7f026957f99a058848b02d1f451"}
Nov 03 14:50:17.926346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.926296    6481 scope.go:122] "RemoveContainer" containerID="a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3"
Nov 03 14:50:17.935908 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.935891    6481 scope.go:122] "RemoveContainer" containerID="a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3"
Nov 03 14:50:17.936168 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:50:17.936149    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3\": container with ID starting with a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3 not found: ID does not exist" containerID="a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3"
Nov 03 14:50:17.936297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.936182    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3"} err="failed to get container status \"a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3\": rpc error: code = NotFound desc = could not find container \"a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3\": container with ID starting with a0d3bf1aca1597af34169be70eec1da0842d1669d963f31de33d63083b5e50b3 not found: ID does not exist"
Nov 03 14:50:17.981120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.981087    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"]
Nov 03 14:50:17.987654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:17.987630    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-7458dc747c-gcb6d"]
Nov 03 14:50:18.066015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:18.065948    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ecf21f3-c878-405d-9684-86c2008d43d1" path="/var/lib/kubelet/pods/4ecf21f3-c878-405d-9684-86c2008d43d1/volumes"
Nov 03 14:50:18.356896 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:18.356830    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:18.576436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:18.576409    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:22.211923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:22.211890    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:22.252654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:22.252624    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:50:22.252825 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:22.252727    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:50:22.430148 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:22.430116    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:23.842414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:23.842387    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:24.075030 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:24.074996    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:26.009940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:26.009907    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:27.868179 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:27.868144    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:50:27.990067 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:50:27.990031    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:50:28.090126 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:28.090088    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:30.114858 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:30.114822    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:32.226004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:32.225973    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:32.253184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:32.253156    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:50:34.337620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:34.337588    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:36.424136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:36.424105    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:37.899942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:37.899909    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:38.427826 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:38.427796    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:38.636109 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:38.636072    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:40.523007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:40.522965    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:40.844002 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:40.843933    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:42.252672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:42.252641    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 14:50:42.628923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:42.628845    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:43.105701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:43.105662    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:50:44.703163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:44.703131    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:46.701862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:46.701830    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:47.261540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.261513    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:50:47.419864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.419831    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f\" (UniqueName: \"kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f\") pod \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") "
Nov 03 14:50:47.420046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.419893    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs\") pod \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\" (UID: \"0e8f6cf1-f473-4e21-8027-19b20f8a868d\") "
Nov 03 14:50:47.421994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.421965    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs" pod "0e8f6cf1-f473-4e21-8027-19b20f8a868d" (UID: "0e8f6cf1-f473-4e21-8027-19b20f8a868d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:50:47.422125 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.422024    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f" pod "0e8f6cf1-f473-4e21-8027-19b20f8a868d" (UID: "0e8f6cf1-f473-4e21-8027-19b20f8a868d"). InnerVolumeSpecName "kube-api-access-tb99f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:50:47.521768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.521687    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb99f\" (UniqueName: \"kubernetes.io/projected/0e8f6cf1-f473-4e21-8027-19b20f8a868d-kube-api-access-tb99f\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:47.521768 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:47.521718    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8f6cf1-f473-4e21-8027-19b20f8a868d-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:50:48.095473 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.095444    6481 generic.go:356] "Generic (PLEG): container finished" podID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerID="a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9" exitCode=0
Nov 03 14:50:48.095913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.095499    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"
Nov 03 14:50:48.095913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.095536    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" event={"ID":"0e8f6cf1-f473-4e21-8027-19b20f8a868d","Type":"ContainerDied","Data":"a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9"}
Nov 03 14:50:48.095913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.095590    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m" event={"ID":"0e8f6cf1-f473-4e21-8027-19b20f8a868d","Type":"ContainerDied","Data":"4f65de205abfd3d00a4d5768ffba85a8888bb255df38a55269dcea678166ed33"}
Nov 03 14:50:48.095913 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.095617    6481 scope.go:122] "RemoveContainer" containerID="a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9"
Nov 03 14:50:48.104593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.104398    6481 scope.go:122] "RemoveContainer" containerID="a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9"
Nov 03 14:50:48.104686 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:50:48.104665    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9\": container with ID starting with a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9 not found: ID does not exist" containerID="a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9"
Nov 03 14:50:48.104731 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.104694    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9"} err="failed to get container status \"a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9\": rpc error: code = NotFound desc = could not find container \"a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9\": container with ID starting with a289e7eb498c501a6316b7b22967caae0aceba592baa238e30858a3490e60fe9 not found: ID does not exist"
Nov 03 14:50:48.146853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.146831    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"]
Nov 03 14:50:48.158592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.158571    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-ea8adfc6-kserve-router-scheduler-c697cb76-mtj4m"]
Nov 03 14:50:48.678194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:48.678155    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:50.065373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:50.065340    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" path="/var/lib/kubelet/pods/0e8f6cf1-f473-4e21-8027-19b20f8a868d/volumes"
Nov 03 14:50:50.598381 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:50.598265    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:52.609122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:52.609090    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:54.603161 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:54.603112    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:56.683458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:56.683420    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:57.986428 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:50:57.986382    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:50:58.710485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:58.710451    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:50:59.712065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:50:59.712030    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:51:00.799326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:00.799296    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:01.014448 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:01.014417    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:02.900373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:02.900342    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:04.889606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:04.889574    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:06.894903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:06.894867    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:08.308665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:08.308633    6481 scope.go:122] "RemoveContainer" containerID="b716d52ce9e0bed7349f588e534408ab0e43e34d6a9d5e91bc9d020439680f54"
Nov 03 14:51:08.317144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:08.317122    6481 scope.go:122] "RemoveContainer" containerID="70b878d8dcb8ca6ca9d18f47327d67b6b3eade1cdf73520e65e29783e2c91a24"
Nov 03 14:51:08.325046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:08.325028    6481 scope.go:122] "RemoveContainer" containerID="ae42cb848b6abb1114109e290a469a63b98879d1b154d0d9c9347879c35f3cc0"
Nov 03 14:51:08.335488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:08.335468    6481 scope.go:122] "RemoveContainer" containerID="6f1f99911dc9ab1796c3cf85ccf0a3ce6e23ff7c5ddd5a75c38b500d7ce2d538"
Nov 03 14:51:08.810371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:08.810337    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:10.815547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:10.815511    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:12.522747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.522709    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"]
Nov 03 14:51:12.523317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.523296    6481 cpu_manager.go:414] "containerMap: removing container" podUID="4ecf21f3-c878-405d-9684-86c2008d43d1" containerName="main"
Nov 03 14:51:12.523437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.523318    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ecf21f3-c878-405d-9684-86c2008d43d1" containerName="main"
Nov 03 14:51:12.523437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.523355    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main"
Nov 03 14:51:12.523437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.523363    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main"
Nov 03 14:51:12.523595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.523498    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ecf21f3-c878-405d-9684-86c2008d43d1" containerName="main"
Nov 03 14:51:12.523595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.523516    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8f6cf1-f473-4e21-8027-19b20f8a868d" containerName="main"
Nov 03 14:51:12.528864 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.528842    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.548334 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.548311    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"]
Nov 03 14:51:12.557367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.557347    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:51:12.557367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.557351    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:51:12.557536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.557370    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:51:12.557536 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.557349    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:51:12.666447 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.666420    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.666606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.666480    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lcc\" (UniqueName: \"kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.666679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.666632    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.767773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.767736    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.767957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.767809    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.767957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.767897    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lcc\" (UniqueName: \"kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.768438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.768418    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.770159 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.770138    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.785999 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.785933    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:12.789478 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.789456    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lcc\" (UniqueName: \"kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc\") pod \"maas-api-revoke-sub-9ea085b5-z4b5f\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:12.842080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:12.842056    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:13.139504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:13.139477    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"]
Nov 03 14:51:13.141169 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:51:13.141139    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409d059a_c43a_4a3f_b76b_d46ef991eb01.slice/crio-c790d76f15969148fc6c69f4ae9461ec2f95fb93c8a2c408b48aa5bc0bd5b0fc WatchSource:0}: Error finding container c790d76f15969148fc6c69f4ae9461ec2f95fb93c8a2c408b48aa5bc0bd5b0fc: Status 404 returned error can't find the container with id c790d76f15969148fc6c69f4ae9461ec2f95fb93c8a2c408b48aa5bc0bd5b0fc
Nov 03 14:51:13.235098 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:13.235056    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f" event={"ID":"409d059a-c43a-4a3f-b76b-d46ef991eb01","Type":"ContainerStarted","Data":"c790d76f15969148fc6c69f4ae9461ec2f95fb93c8a2c408b48aa5bc0bd5b0fc"}
Nov 03 14:51:14.243583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:14.243544    6481 generic.go:356] "Generic (PLEG): container finished" podID="409d059a-c43a-4a3f-b76b-d46ef991eb01" containerID="c1fc803e33d1862b6e45a874f5b4d9829031d863f5ef9b158d5a1bdef635a9e1" exitCode=0
Nov 03 14:51:14.244026 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:14.243627    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f" event={"ID":"409d059a-c43a-4a3f-b76b-d46ef991eb01","Type":"ContainerDied","Data":"c1fc803e33d1862b6e45a874f5b4d9829031d863f5ef9b158d5a1bdef635a9e1"}
Nov 03 14:51:14.895465 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:14.895429    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:15.382810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.382788    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:15.511672 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.511602    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca\") pod \"409d059a-c43a-4a3f-b76b-d46ef991eb01\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") "
Nov 03 14:51:15.511892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.511683    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca\") pod \"409d059a-c43a-4a3f-b76b-d46ef991eb01\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") "
Nov 03 14:51:15.511892 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.511777    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc\" (UniqueName: \"kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc\") pod \"409d059a-c43a-4a3f-b76b-d46ef991eb01\" (UID: \"409d059a-c43a-4a3f-b76b-d46ef991eb01\") "
Nov 03 14:51:15.512025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.512000    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca" pod "409d059a-c43a-4a3f-b76b-d46ef991eb01" (UID: "409d059a-c43a-4a3f-b76b-d46ef991eb01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:51:15.513695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.513666    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca" pod "409d059a-c43a-4a3f-b76b-d46ef991eb01" (UID: "409d059a-c43a-4a3f-b76b-d46ef991eb01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:51:15.513828 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.513772    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc" pod "409d059a-c43a-4a3f-b76b-d46ef991eb01" (UID: "409d059a-c43a-4a3f-b76b-d46ef991eb01"). InnerVolumeSpecName "kube-api-access-r6lcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:51:15.613430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.613397    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:51:15.613430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.613428    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6lcc\" (UniqueName: \"kubernetes.io/projected/409d059a-c43a-4a3f-b76b-d46ef991eb01-kube-api-access-r6lcc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:51:15.613601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:15.613442    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/409d059a-c43a-4a3f-b76b-d46ef991eb01-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:51:16.254701 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:16.254676    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"
Nov 03 14:51:16.254860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:16.254697    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f" event={"ID":"409d059a-c43a-4a3f-b76b-d46ef991eb01","Type":"ContainerDied","Data":"c790d76f15969148fc6c69f4ae9461ec2f95fb93c8a2c408b48aa5bc0bd5b0fc"}
Nov 03 14:51:16.254860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:16.254739    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c790d76f15969148fc6c69f4ae9461ec2f95fb93c8a2c408b48aa5bc0bd5b0fc"
Nov 03 14:51:16.884777 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:16.884746    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:18.892779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:18.892741    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:20.892513 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:20.892479    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:22.786407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:22.786374    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:23.471793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:23.471760    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:23.956587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:23.956555    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:24.096418 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:24.096391    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:51:24.815607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:24.815569    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:26.886340 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:26.886309    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:27.985581 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:51:27.985546    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:51:28.564184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:28.564150    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:28.990132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:28.990103    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:31.008946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:31.008912    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:31.490488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:31.490461    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:31.699511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:31.699466    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:31.913905 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:31.913877    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:33.009869 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:33.009829    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:35.425555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:35.425520    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:51:38.808782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:38.808747    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:39.025096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:39.025063    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:54.966391 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:51:54.966357    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:51:57.985210 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:51:57.985173    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:52:01.053724 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:01.053691    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:03.042870 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:03.042835    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"]
Nov 03 14:52:03.054141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:03.054107    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qrsvf"]
Nov 03 14:52:03.678640 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:03.678606    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:03.893883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:03.893836    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:04.066283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:04.066156    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd224be-badc-4e15-a2bc-c07ab287963c" path="/var/lib/kubelet/pods/7bd224be-badc-4e15-a2bc-c07ab287963c/volumes"
Nov 03 14:52:05.323412 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.323377    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:05.540195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.540159    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:05.600486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.600416    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"]
Nov 03 14:52:05.600817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.600805    6481 cpu_manager.go:414] "containerMap: removing container" podUID="409d059a-c43a-4a3f-b76b-d46ef991eb01" containerName="revoke-keys"
Nov 03 14:52:05.600862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.600818    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="409d059a-c43a-4a3f-b76b-d46ef991eb01" containerName="revoke-keys"
Nov 03 14:52:05.600921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.600911    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="409d059a-c43a-4a3f-b76b-d46ef991eb01" containerName="revoke-keys"
Nov 03 14:52:05.604267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.604251    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.633291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.633267    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-a58e3e\""
Nov 03 14:52:05.633422 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.633299    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-a7a5aa60-dockercfg-kpxp4\""
Nov 03 14:52:05.635758 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.635735    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"]
Nov 03 14:52:05.690891 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.690856    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"]
Nov 03 14:52:05.695572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.695545    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.723072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.723033    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"]
Nov 03 14:52:05.754595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.754570    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:05.789763 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.789733    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lg8q\" (UniqueName: \"kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.789909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.789801    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5bhb\" (UniqueName: \"kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.789909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.789891    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.790003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.789920    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.890812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.890742    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lg8q\" (UniqueName: \"kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.890812 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.890799    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5bhb\" (UniqueName: \"kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.890982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.890842    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.890982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.890865    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.891568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.891549    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.891568 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.891559    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.912332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.912305    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lg8q\" (UniqueName: \"kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q\") pod \"payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:05.912439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.912359    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5bhb\" (UniqueName: \"kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb\") pod \"payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:05.917296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:05.917280    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:06.030003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.029970    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:06.220938 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.220913    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"]
Nov 03 14:52:06.221883 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:52:06.221845    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb11ed41_85d5_4a60_9b8a_c5a03ee89fb8.slice/crio-f5aa5ba6d8147f26d6e24ed823e7d4fd7211a9238c492d82f042ee1f142b7924 WatchSource:0}: Error finding container f5aa5ba6d8147f26d6e24ed823e7d4fd7211a9238c492d82f042ee1f142b7924: Status 404 returned error can't find the container with id f5aa5ba6d8147f26d6e24ed823e7d4fd7211a9238c492d82f042ee1f142b7924
Nov 03 14:52:06.330572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.330545    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"]
Nov 03 14:52:06.332335 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:52:06.332309    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c0f6ae_1b0d_4adb_a274_390fd32bcfc2.slice/crio-fad245fbb473ca766625dcee4eef34fedd01a9340fed684796423bef7081c0eb WatchSource:0}: Error finding container fad245fbb473ca766625dcee4eef34fedd01a9340fed684796423bef7081c0eb: Status 404 returned error can't find the container with id fad245fbb473ca766625dcee4eef34fedd01a9340fed684796423bef7081c0eb
Nov 03 14:52:06.512264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.512164    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj" event={"ID":"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8","Type":"ContainerStarted","Data":"a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59"}
Nov 03 14:52:06.512264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.512218    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj" event={"ID":"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8","Type":"ContainerStarted","Data":"f5aa5ba6d8147f26d6e24ed823e7d4fd7211a9238c492d82f042ee1f142b7924"}
Nov 03 14:52:06.512264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.512255    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:06.513793 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.513768    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k" event={"ID":"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2","Type":"ContainerStarted","Data":"77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6"}
Nov 03 14:52:06.513922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.513807    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k" event={"ID":"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2","Type":"ContainerStarted","Data":"fad245fbb473ca766625dcee4eef34fedd01a9340fed684796423bef7081c0eb"}
Nov 03 14:52:06.513922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.513849    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:06.553782 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.553737    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj" podStartSLOduration=1.553723821 podStartE2EDuration="1.553723821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:06.552380932 +0000 UTC m=+2404.602412129" watchObservedRunningTime="2026-11-03 14:52:06.553723821 +0000 UTC m=+2404.603755004"
Nov 03 14:52:06.596247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:06.596185    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k" podStartSLOduration=1.596174011 podStartE2EDuration="1.596174011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:06.594668832 +0000 UTC m=+2404.644700027" watchObservedRunningTime="2026-11-03 14:52:06.596174011 +0000 UTC m=+2404.646205193"
Nov 03 14:52:07.706679 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:07.706644    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:07.928240 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:07.928185    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:09.273044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:09.273013    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:09.600889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:09.600807    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:52:09.852333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:09.851006    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:11.892162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:11.892132    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:13.903462 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:13.903424    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:15.499481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.499446    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"]
Nov 03 14:52:15.503380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.503360    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.530118 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.530076    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:52:15.530259 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.530159    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:52:15.530587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.530568    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:52:15.530687 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.530594    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:52:15.533304 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.533269    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"]
Nov 03 14:52:15.594140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.594115    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.594336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.594182    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.594336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.594310    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6484\" (UniqueName: \"kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.695041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.695014    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.695185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.695077    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.695185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.695104    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6484\" (UniqueName: \"kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.695739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.695715    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.697444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.697419    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.716088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.716067    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6484\" (UniqueName: \"kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.816924 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.816868    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:15.983996 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:15.983963    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:16.123994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:16.123973    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"]
Nov 03 14:52:16.126242 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:52:16.126204    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdeb462d_92ce_4a6d_adf4_e9ad49e5ce49.slice/crio-7912a5cffa71b44dbb830723a295251cd1f94a00faa9709fd19bbaafefcbc311 WatchSource:0}: Error finding container 7912a5cffa71b44dbb830723a295251cd1f94a00faa9709fd19bbaafefcbc311: Status 404 returned error can't find the container with id 7912a5cffa71b44dbb830723a295251cd1f94a00faa9709fd19bbaafefcbc311
Nov 03 14:52:16.574937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:16.574901    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f" event={"ID":"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49","Type":"ContainerStarted","Data":"685cfcfafa9de9227e9716cc07690f899ed9cde09db95dca5bf936a625d830d6"}
Nov 03 14:52:16.575446 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:16.574961    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f" event={"ID":"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49","Type":"ContainerStarted","Data":"7912a5cffa71b44dbb830723a295251cd1f94a00faa9709fd19bbaafefcbc311"}
Nov 03 14:52:16.618630 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:16.618583    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f" podStartSLOduration=1.618569737 podStartE2EDuration="1.618569737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:16.617870877 +0000 UTC m=+2414.667902060" watchObservedRunningTime="2026-11-03 14:52:16.618569737 +0000 UTC m=+2414.668600919"
Nov 03 14:52:16.667814 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:16.667784    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:16.877626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:16.877560    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:17.532594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:17.532566    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:17.532774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:17.532630    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:17.583181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:17.583136    6481 generic.go:356] "Generic (PLEG): container finished" podID="cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" containerID="685cfcfafa9de9227e9716cc07690f899ed9cde09db95dca5bf936a625d830d6" exitCode=0
Nov 03 14:52:17.583721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:17.583285    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f" event={"ID":"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49","Type":"ContainerDied","Data":"685cfcfafa9de9227e9716cc07690f899ed9cde09db95dca5bf936a625d830d6"}
Nov 03 14:52:17.888653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:17.888571    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:18.717431 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.717406    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:18.719990 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.719969    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca\") pod \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") "
Nov 03 14:52:18.720141 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.720127    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca\") pod \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") "
Nov 03 14:52:18.720184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.720157    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484\" (UniqueName: \"kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484\") pod \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\" (UID: \"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49\") "
Nov 03 14:52:18.720332 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.720311    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca" pod "cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" (UID: "cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:52:18.720526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.720511    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:18.722146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.722126    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484" pod "cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" (UID: "cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49"). InnerVolumeSpecName "kube-api-access-g6484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:52:18.722202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.722129    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca" pod "cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" (UID: "cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:52:18.821786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.821754    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:18.821786 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:18.821782    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6484\" (UniqueName: \"kubernetes.io/projected/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49-kube-api-access-g6484\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:19.593707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:19.593679    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"
Nov 03 14:52:19.593903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:19.593686    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f" event={"ID":"cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49","Type":"ContainerDied","Data":"7912a5cffa71b44dbb830723a295251cd1f94a00faa9709fd19bbaafefcbc311"}
Nov 03 14:52:19.593903 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:19.593795    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7912a5cffa71b44dbb830723a295251cd1f94a00faa9709fd19bbaafefcbc311"
Nov 03 14:52:19.881365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:19.881286    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:21.892099 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:21.892065    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:23.768074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:23.768043    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:25.654873 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:25.654841    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:25.990104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:25.990071    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"]
Nov 03 14:52:26.027992 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:26.027963    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"]
Nov 03 14:52:26.251766 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:26.251675    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:26.466657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:26.466620    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:27.481883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:27.481841    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:27.985766 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:52:27.985719    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:52:29.063264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:29.063214    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:29.645511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:29.645475    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:29.784194 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:29.784159    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"]
Nov 03 14:52:29.795540 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:29.795511    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a58e3e-gjr4f"]
Nov 03 14:52:29.860315 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:29.860286    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:30.066504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:30.066475    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" path="/var/lib/kubelet/pods/cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49/volumes"
Nov 03 14:52:30.332866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:30.332785    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:30.991283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:30.991224    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k" podUID="b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" containerName="payload-processing" containerID="cri-o://77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6" gracePeriod=25
Nov 03 14:52:31.028852 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.028812    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj" podUID="db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" containerName="payload-pre-processing" containerID="cri-o://a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59" gracePeriod=25
Nov 03 14:52:31.283829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.283803    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:31.301081 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.301054    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:31.343829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.343792    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume\") pod \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") "
Nov 03 14:52:31.343963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.343856    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q\" (UniqueName: \"kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q\") pod \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\" (UID: \"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8\") "
Nov 03 14:52:31.343963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.343883    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume\") pod \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") "
Nov 03 14:52:31.343963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.343941    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb\" (UniqueName: \"kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb\") pod \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\" (UID: \"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2\") "
Nov 03 14:52:31.344177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.344156    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume" pod "db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" (UID: "db11ed41-85d5-4a60-9b8a-c5a03ee89fb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:52:31.344343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.344319    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume" pod "b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" (UID: "b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:52:31.344404 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.344349    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:31.346032 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.346004    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb" pod "b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" (UID: "b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2"). InnerVolumeSpecName "kube-api-access-v5bhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:52:31.346110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.346055    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q" pod "db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" (UID: "db11ed41-85d5-4a60-9b8a-c5a03ee89fb8"). InnerVolumeSpecName "kube-api-access-8lg8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:52:31.447579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.447490    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5bhb\" (UniqueName: \"kubernetes.io/projected/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-kube-api-access-v5bhb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:31.447579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.447527    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lg8q\" (UniqueName: \"kubernetes.io/projected/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8-kube-api-access-8lg8q\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:31.447579 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.447543    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:52:31.660880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.660845    6481 generic.go:356] "Generic (PLEG): container finished" podID="b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" containerID="77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6" exitCode=0
Nov 03 14:52:31.661079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.660901    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"
Nov 03 14:52:31.661079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.660929    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k" event={"ID":"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2","Type":"ContainerDied","Data":"77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6"}
Nov 03 14:52:31.661079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.660985    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k" event={"ID":"b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2","Type":"ContainerDied","Data":"fad245fbb473ca766625dcee4eef34fedd01a9340fed684796423bef7081c0eb"}
Nov 03 14:52:31.661079 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.661009    6481 scope.go:122] "RemoveContainer" containerID="77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6"
Nov 03 14:52:31.662188 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.662167    6481 generic.go:356] "Generic (PLEG): container finished" podID="db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" containerID="a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59" exitCode=0
Nov 03 14:52:31.662314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.662214    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"
Nov 03 14:52:31.662314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.662270    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj" event={"ID":"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8","Type":"ContainerDied","Data":"a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59"}
Nov 03 14:52:31.662438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.662330    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj" event={"ID":"db11ed41-85d5-4a60-9b8a-c5a03ee89fb8","Type":"ContainerDied","Data":"f5aa5ba6d8147f26d6e24ed823e7d4fd7211a9238c492d82f042ee1f142b7924"}
Nov 03 14:52:31.669694 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.669675    6481 scope.go:122] "RemoveContainer" containerID="77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6"
Nov 03 14:52:31.669957 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:52:31.669939    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6\": container with ID starting with 77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6 not found: ID does not exist" containerID="77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6"
Nov 03 14:52:31.670018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.669964    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6"} err="failed to get container status \"77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6\": rpc error: code = NotFound desc = could not find container \"77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6\": container with ID starting with 77c1845be674a2d9682d130b8ba1d70ca583495b51a4baf21dca6940bb668ef6 not found: ID does not exist"
Nov 03 14:52:31.670018 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.669981    6481 scope.go:122] "RemoveContainer" containerID="a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59"
Nov 03 14:52:31.678367 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.678351    6481 scope.go:122] "RemoveContainer" containerID="a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59"
Nov 03 14:52:31.678591 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:52:31.678573    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59\": container with ID starting with a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59 not found: ID does not exist" containerID="a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59"
Nov 03 14:52:31.678654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.678596    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59"} err="failed to get container status \"a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59\": rpc error: code = NotFound desc = could not find container \"a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59\": container with ID starting with a128f55bd6c9c6e24878434caec1943fc5585d7a0eae786e3eea9e5031ac2f59 not found: ID does not exist"
Nov 03 14:52:31.713224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.713201    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"]
Nov 03 14:52:31.720408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.720390    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-a58e3e-587c5vljrj"]
Nov 03 14:52:31.760202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.760180    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"]
Nov 03 14:52:31.767549 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:31.767524    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-a58e3e-5b74b684cmj42k"]
Nov 03 14:52:32.065658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:32.065629    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" path="/var/lib/kubelet/pods/b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2/volumes"
Nov 03 14:52:32.066052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:32.066036    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" path="/var/lib/kubelet/pods/db11ed41-85d5-4a60-9b8a-c5a03ee89fb8/volumes"
Nov 03 14:52:32.079156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:32.079133    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:52:35.770937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.770895    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"]
Nov 03 14:52:35.771529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771508    6481 cpu_manager.go:414] "containerMap: removing container" podUID="cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" containerName="revoke-keys"
Nov 03 14:52:35.771529 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771529    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" containerName="revoke-keys"
Nov 03 14:52:35.771702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771569    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" containerName="payload-processing"
Nov 03 14:52:35.771702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771577    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" containerName="payload-processing"
Nov 03 14:52:35.771702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771608    6481 cpu_manager.go:414] "containerMap: removing container" podUID="db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" containerName="payload-pre-processing"
Nov 03 14:52:35.771702 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771616    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" containerName="payload-pre-processing"
Nov 03 14:52:35.771909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771709    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdeb462d-92ce-4a6d-adf4-e9ad49e5ce49" containerName="revoke-keys"
Nov 03 14:52:35.771909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771727    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3c0f6ae-1b0d-4adb-a274-390fd32bcfc2" containerName="payload-processing"
Nov 03 14:52:35.771909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.771740    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="db11ed41-85d5-4a60-9b8a-c5a03ee89fb8" containerName="payload-pre-processing"
Nov 03 14:52:35.776243 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.776207    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.793699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.793678    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"]
Nov 03 14:52:35.803178 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.803159    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0e21d4af-kserve-self-signed-certs\""
Nov 03 14:52:35.892088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.892060    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.892217 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.892105    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.892307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.892220    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.892307 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.892287    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.892382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.892358    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knk48\" (UniqueName: \"kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.892437 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.892423    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993477 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993435    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993494    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993512    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knk48\" (UniqueName: \"kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993540    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993592    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993680 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993614    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993888    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.993975 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.993948    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.994050 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.994019    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.995726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.995707    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:35.996039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:35.996024    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:36.017539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.017512    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knk48\" (UniqueName: \"kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48\") pod \"e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") " pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:36.090252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.090170    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:36.205933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.205906    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:36.392889 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.392866    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"]
Nov 03 14:52:36.394623 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:52:36.394595    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod226c6820_71f4_4b03_92a7_2c524a77e9da.slice/crio-e97e08baadd836a9b4548c7522aeb94509fc0879881159a7bbc8ecda4c04299f WatchSource:0}: Error finding container e97e08baadd836a9b4548c7522aeb94509fc0879881159a7bbc8ecda4c04299f: Status 404 returned error can't find the container with id e97e08baadd836a9b4548c7522aeb94509fc0879881159a7bbc8ecda4c04299f
Nov 03 14:52:36.416407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.416377    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:36.692403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.692367    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td" event={"ID":"226c6820-71f4-4b03-92a7-2c524a77e9da","Type":"ContainerStarted","Data":"bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809"}
Nov 03 14:52:36.692403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.692410    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td" event={"ID":"226c6820-71f4-4b03-92a7-2c524a77e9da","Type":"ContainerStarted","Data":"e97e08baadd836a9b4548c7522aeb94509fc0879881159a7bbc8ecda4c04299f"}
Nov 03 14:52:36.735326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:36.735274    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td" podStartSLOduration=1.7352605620000001 podStartE2EDuration="1.735260562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:36.733108973 +0000 UTC m=+2434.783140159" watchObservedRunningTime="2026-11-03 14:52:36.735260562 +0000 UTC m=+2434.785291744"
Nov 03 14:52:37.091193 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:37.091121    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:37.368522 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:37.368441    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:52:43.044460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:43.044381    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"]
Nov 03 14:52:43.055670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:43.055641    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-8qgn5"]
Nov 03 14:52:44.065861 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:44.065827    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f4c5c7-650d-4e58-84ee-cf6e990fcec9" path="/var/lib/kubelet/pods/a2f4c5c7-650d-4e58-84ee-cf6e990fcec9/volumes"
Nov 03 14:52:46.091038 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:46.091009    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:46.103399 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:46.103375    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:46.796255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:46.796213    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:52:53.662131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.662090    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"]
Nov 03 14:52:53.665594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.665575    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.693588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.693567    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-0e21d4af-kserve-self-signed-certs\""
Nov 03 14:52:53.695526 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.695499    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"]
Nov 03 14:52:53.814902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.814871    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:53.842276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.842218    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.842460 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.842358    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.842528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.842483    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.842627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.842597    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvw5n\" (UniqueName: \"kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.842844 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.842822    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.842923 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.842869    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944201    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944306 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944285    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944335    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944378    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvw5n\" (UniqueName: \"kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944412    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944519 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944443    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944729 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944683    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944734    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.944789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.944765    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.946649 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.946627    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.946765 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.946730    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.964721 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.964697    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvw5n\" (UniqueName: \"kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n\") pod \"e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") " pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:53.980198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:53.980172    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:54.280043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:54.280021    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"]
Nov 03 14:52:54.281660 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:52:54.281630    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c323af7_589b_414c_9296_e644f08505ee.slice/crio-589ae097f4a08089aab548f9e0148746587e6955bd803cd9d2da7b4a0f912dc4 WatchSource:0}: Error finding container 589ae097f4a08089aab548f9e0148746587e6955bd803cd9d2da7b4a0f912dc4: Status 404 returned error can't find the container with id 589ae097f4a08089aab548f9e0148746587e6955bd803cd9d2da7b4a0f912dc4
Nov 03 14:52:54.829982 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:54.829947    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r" event={"ID":"4c323af7-589b-414c-9296-e644f08505ee","Type":"ContainerStarted","Data":"a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261"}
Nov 03 14:52:54.830398 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:54.829990    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r" event={"ID":"4c323af7-589b-414c-9296-e644f08505ee","Type":"ContainerStarted","Data":"589ae097f4a08089aab548f9e0148746587e6955bd803cd9d2da7b4a0f912dc4"}
Nov 03 14:52:54.872692 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:54.872645    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r" podStartSLOduration=1.872632047 podStartE2EDuration="1.872632047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:54.871882199 +0000 UTC m=+2452.921913383" watchObservedRunningTime="2026-11-03 14:52:54.872632047 +0000 UTC m=+2452.922663230"
Nov 03 14:52:54.981033 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:54.981005    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:52:56.112685 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:56.112647    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:52:56.139407 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:56.139382    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:56.351187 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:52:56.351137    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:52:57.998948 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:52:57.998893    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:53:01.306821 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:01.306775    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:53:03.980485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:03.980455    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:53:03.992618 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:03.992594    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:53:04.939481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:04.939455    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:53:08.385657 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:08.385627    6481 scope.go:122] "RemoveContainer" containerID="657dcdd1c19028f3ee215dcfac0015bab30def7b9aeed857a094795468720cee"
Nov 03 14:53:08.394389 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:08.394369    6481 scope.go:122] "RemoveContainer" containerID="118b4975bb5eb001cc96f9584aa0dc057c032d08e8e6cfd36f1532d4708f03ed"
Nov 03 14:53:09.632695 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.632662    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"]
Nov 03 14:53:09.637854 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.637837    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.673317 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.673286    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"]
Nov 03 14:53:09.673495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.673474    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:53:09.673774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.673754    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:53:09.673902 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.673886    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:53:09.674059 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.674041    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:53:09.674196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.674179    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.674321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.674288    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfpsn\" (UniqueName: \"kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.674426 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.674409    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.775333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.775302    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfpsn\" (UniqueName: \"kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.775452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.775371    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.775452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.775412    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.776015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.775996    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.777645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.777620    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.797057 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.797029    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfpsn\" (UniqueName: \"kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn\") pod \"maas-api-revoke-sub-452b6d7b-wphgw\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:09.949946 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:09.949920    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:10.005094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:10.005060    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:53:10.253674 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:10.253652    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"]
Nov 03 14:53:10.255398 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:53:10.255368    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6192ecf5_b450_4ff5_a9c8_efe4c72255c9.slice/crio-fe1efb0e59d4024e734b0b3efce5644bca646ead44118b9ed609fa6d62d3c667 WatchSource:0}: Error finding container fe1efb0e59d4024e734b0b3efce5644bca646ead44118b9ed609fa6d62d3c667: Status 404 returned error can't find the container with id fe1efb0e59d4024e734b0b3efce5644bca646ead44118b9ed609fa6d62d3c667
Nov 03 14:53:10.963280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:10.963219    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw" event={"ID":"6192ecf5-b450-4ff5-a9c8-efe4c72255c9","Type":"ContainerStarted","Data":"00b7b9476b3fcdafe5bbe94bb237f771e5f82f0691f1f4a6d665d098869555a5"}
Nov 03 14:53:10.963933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:10.963891    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw" event={"ID":"6192ecf5-b450-4ff5-a9c8-efe4c72255c9","Type":"ContainerStarted","Data":"fe1efb0e59d4024e734b0b3efce5644bca646ead44118b9ed609fa6d62d3c667"}
Nov 03 14:53:11.008620 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:11.008557    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw" podStartSLOduration=2.008537026 podStartE2EDuration="2.008537026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:53:11.005125384 +0000 UTC m=+2469.055156568" watchObservedRunningTime="2026-11-03 14:53:11.008537026 +0000 UTC m=+2469.058568210"
Nov 03 14:53:11.995282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:11.995208    6481 generic.go:356] "Generic (PLEG): container finished" podID="6192ecf5-b450-4ff5-a9c8-efe4c72255c9" containerID="00b7b9476b3fcdafe5bbe94bb237f771e5f82f0691f1f4a6d665d098869555a5" exitCode=0
Nov 03 14:53:11.995995 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:11.995968    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw" event={"ID":"6192ecf5-b450-4ff5-a9c8-efe4c72255c9","Type":"ContainerDied","Data":"00b7b9476b3fcdafe5bbe94bb237f771e5f82f0691f1f4a6d665d098869555a5"}
Nov 03 14:53:13.841071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:13.840740    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:14.013311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.003433    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca\") pod \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") "
Nov 03 14:53:14.013311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.003519    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn\" (UniqueName: \"kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn\") pod \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") "
Nov 03 14:53:14.013311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.003742    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca\") pod \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\" (UID: \"6192ecf5-b450-4ff5-a9c8-efe4c72255c9\") "
Nov 03 14:53:14.071866 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.071487    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca" pod "6192ecf5-b450-4ff5-a9c8-efe4c72255c9" (UID: "6192ecf5-b450-4ff5-a9c8-efe4c72255c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:53:14.128293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.128213    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:14.182165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.182109    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca" pod "6192ecf5-b450-4ff5-a9c8-efe4c72255c9" (UID: "6192ecf5-b450-4ff5-a9c8-efe4c72255c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:53:14.201921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.201566    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw" event={"ID":"6192ecf5-b450-4ff5-a9c8-efe4c72255c9","Type":"ContainerDied","Data":"fe1efb0e59d4024e734b0b3efce5644bca646ead44118b9ed609fa6d62d3c667"}
Nov 03 14:53:14.201921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.201630    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe1efb0e59d4024e734b0b3efce5644bca646ead44118b9ed609fa6d62d3c667"
Nov 03 14:53:14.201921 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.201704    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"
Nov 03 14:53:14.202887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.202844    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn" pod "6192ecf5-b450-4ff5-a9c8-efe4c72255c9" (UID: "6192ecf5-b450-4ff5-a9c8-efe4c72255c9"). InnerVolumeSpecName "kube-api-access-nfpsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:53:14.230136 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.230113    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:14.230361 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:14.230346    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfpsn\" (UniqueName: \"kubernetes.io/projected/6192ecf5-b450-4ff5-a9c8-efe4c72255c9-kube-api-access-nfpsn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:16.046575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:16.046539    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"]
Nov 03 14:53:16.058423 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:16.058398    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-k5frq"]
Nov 03 14:53:16.067129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:16.067097    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61557c54-a689-4e12-8c38-91db501f915c" path="/var/lib/kubelet/pods/61557c54-a689-4e12-8c38-91db501f915c/volumes"
Nov 03 14:53:20.910111 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:20.910077    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"]
Nov 03 14:53:21.050994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:21.050960    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:53:23.153890 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:23.153844    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:53:27.014254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:26.997462    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"]
Nov 03 14:53:27.199545 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:27.199505    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:53:28.005940 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:53:28.005901    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:53:28.558810 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:28.558778    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:53:28.779022 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:28.778988    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:53:30.169984 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.169950    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:53:30.904759 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.904721    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"]
Nov 03 14:53:30.905457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.905432    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6192ecf5-b450-4ff5-a9c8-efe4c72255c9" containerName="revoke-keys"
Nov 03 14:53:30.905457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.905458    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6192ecf5-b450-4ff5-a9c8-efe4c72255c9" containerName="revoke-keys"
Nov 03 14:53:30.905659 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.905559    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6192ecf5-b450-4ff5-a9c8-efe4c72255c9" containerName="revoke-keys"
Nov 03 14:53:30.909162 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.909140    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:30.932428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.932401    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"]
Nov 03 14:53:30.941734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.941563    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:53:30.941734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.941601    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:53:30.941734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.941666    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:53:30.941734 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.941682    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:53:30.953290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.953252    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:30.953432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.953311    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:30.953432 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:30.953410    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqpl\" (UniqueName: \"kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.054605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.054555    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.054816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.054618    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.054816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.054723    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqpl\" (UniqueName: \"kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.055790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.055759    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.057733 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.057708    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.076592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.076565    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqpl\" (UniqueName: \"kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl\") pod \"maas-api-revoke-sub-3e31a506-sbtxt\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.231318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.231279    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:31.568472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:31.568444    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"]
Nov 03 14:53:31.571353 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:53:31.571320    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5882cd8d_1423_434d_9538_5e35ad117da9.slice/crio-8c405167af5d4236dd32e6b833d5682b7d21e2835e9c3b29a6f24550fc52fc44 WatchSource:0}: Error finding container 8c405167af5d4236dd32e6b833d5682b7d21e2835e9c3b29a6f24550fc52fc44: Status 404 returned error can't find the container with id 8c405167af5d4236dd32e6b833d5682b7d21e2835e9c3b29a6f24550fc52fc44
Nov 03 14:53:32.444728 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:32.444691    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt" event={"ID":"5882cd8d-1423-434d-9538-5e35ad117da9","Type":"ContainerStarted","Data":"a2ce352826c822496814d2b357a92cac7b900de20b36aaecea0060ece3102361"}
Nov 03 14:53:32.453523 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:32.453491    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt" event={"ID":"5882cd8d-1423-434d-9538-5e35ad117da9","Type":"ContainerStarted","Data":"8c405167af5d4236dd32e6b833d5682b7d21e2835e9c3b29a6f24550fc52fc44"}
Nov 03 14:53:32.519253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:32.516838    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt" podStartSLOduration=2.51681991 podStartE2EDuration="2.51681991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:53:32.515385457 +0000 UTC m=+2490.565416640" watchObservedRunningTime="2026-11-03 14:53:32.51681991 +0000 UTC m=+2490.566851095"
Nov 03 14:53:33.177252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.170328    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:53:33.177252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.170451    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:53:33.222373 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.222333    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:53:33.222666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.222635    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:53:33.290255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.289328    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:53:33.290255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.289389    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:53:33.500365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.499309    6481 generic.go:356] "Generic (PLEG): container finished" podID="5882cd8d-1423-434d-9538-5e35ad117da9" containerID="a2ce352826c822496814d2b357a92cac7b900de20b36aaecea0060ece3102361" exitCode=0
Nov 03 14:53:33.500365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:33.499427    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt" event={"ID":"5882cd8d-1423-434d-9538-5e35ad117da9","Type":"ContainerDied","Data":"a2ce352826c822496814d2b357a92cac7b900de20b36aaecea0060ece3102361"}
Nov 03 14:53:34.177252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.176351    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:53:34.177252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.176481    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:53:34.691886 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.691852    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:34.767383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.767318    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca\") pod \"5882cd8d-1423-434d-9538-5e35ad117da9\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") "
Nov 03 14:53:34.767383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.767371    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl\" (UniqueName: \"kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl\") pod \"5882cd8d-1423-434d-9538-5e35ad117da9\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") "
Nov 03 14:53:34.767555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.767394    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca\") pod \"5882cd8d-1423-434d-9538-5e35ad117da9\" (UID: \"5882cd8d-1423-434d-9538-5e35ad117da9\") "
Nov 03 14:53:34.767795 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.767775    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca" pod "5882cd8d-1423-434d-9538-5e35ad117da9" (UID: "5882cd8d-1423-434d-9538-5e35ad117da9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:53:34.769419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.769402    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca" pod "5882cd8d-1423-434d-9538-5e35ad117da9" (UID: "5882cd8d-1423-434d-9538-5e35ad117da9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:53:34.769539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.769525    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl" pod "5882cd8d-1423-434d-9538-5e35ad117da9" (UID: "5882cd8d-1423-434d-9538-5e35ad117da9"). InnerVolumeSpecName "kube-api-access-brqpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:53:34.868382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.868357    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:34.868382 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.868380    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqpl\" (UniqueName: \"kubernetes.io/projected/5882cd8d-1423-434d-9538-5e35ad117da9-kube-api-access-brqpl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:34.868554 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:34.868390    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5882cd8d-1423-434d-9538-5e35ad117da9-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:35.514516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:35.514487    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"
Nov 03 14:53:35.514696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:35.514483    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt" event={"ID":"5882cd8d-1423-434d-9538-5e35ad117da9","Type":"ContainerDied","Data":"8c405167af5d4236dd32e6b833d5682b7d21e2835e9c3b29a6f24550fc52fc44"}
Nov 03 14:53:35.514696 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:35.514616    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c405167af5d4236dd32e6b833d5682b7d21e2835e9c3b29a6f24550fc52fc44"
Nov 03 14:53:35.930275 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:35.930215    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td" podUID="226c6820-71f4-4b03-92a7-2c524a77e9da" containerName="main" containerID="cri-o://bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809" gracePeriod=45
Nov 03 14:53:36.181981 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.181926    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:53:36.279658 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279623    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache\") pod \"226c6820-71f4-4b03-92a7-2c524a77e9da\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") "
Nov 03 14:53:36.279809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279676    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs\" (UniqueName: \"kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs\") pod \"226c6820-71f4-4b03-92a7-2c524a77e9da\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") "
Nov 03 14:53:36.279809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279728    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home\") pod \"226c6820-71f4-4b03-92a7-2c524a77e9da\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") "
Nov 03 14:53:36.279809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279759    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm\") pod \"226c6820-71f4-4b03-92a7-2c524a77e9da\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") "
Nov 03 14:53:36.279809 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279802    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir\") pod \"226c6820-71f4-4b03-92a7-2c524a77e9da\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") "
Nov 03 14:53:36.280062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279825    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48\" (UniqueName: \"kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48\") pod \"226c6820-71f4-4b03-92a7-2c524a77e9da\" (UID: \"226c6820-71f4-4b03-92a7-2c524a77e9da\") "
Nov 03 14:53:36.280062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.279906    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache" pod "226c6820-71f4-4b03-92a7-2c524a77e9da" (UID: "226c6820-71f4-4b03-92a7-2c524a77e9da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:36.280062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.280028    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home" pod "226c6820-71f4-4b03-92a7-2c524a77e9da" (UID: "226c6820-71f4-4b03-92a7-2c524a77e9da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:36.280062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.280038    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir" pod "226c6820-71f4-4b03-92a7-2c524a77e9da" (UID: "226c6820-71f4-4b03-92a7-2c524a77e9da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:36.280283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.280177    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:36.280283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.280194    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:36.280283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.280206    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:36.281800 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.281772    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm" pod "226c6820-71f4-4b03-92a7-2c524a77e9da" (UID: "226c6820-71f4-4b03-92a7-2c524a77e9da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:36.281918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.281825    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs" pod "226c6820-71f4-4b03-92a7-2c524a77e9da" (UID: "226c6820-71f4-4b03-92a7-2c524a77e9da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:53:36.281967 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.281929    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48" pod "226c6820-71f4-4b03-92a7-2c524a77e9da" (UID: "226c6820-71f4-4b03-92a7-2c524a77e9da"). InnerVolumeSpecName "kube-api-access-knk48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:53:36.381116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.381081    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/226c6820-71f4-4b03-92a7-2c524a77e9da-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:36.381116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.381110    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/226c6820-71f4-4b03-92a7-2c524a77e9da-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:36.381116 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.381120    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knk48\" (UniqueName: \"kubernetes.io/projected/226c6820-71f4-4b03-92a7-2c524a77e9da-kube-api-access-knk48\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:36.520127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.520046    6481 generic.go:356] "Generic (PLEG): container finished" podID="226c6820-71f4-4b03-92a7-2c524a77e9da" containerID="bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809" exitCode=0
Nov 03 14:53:36.520127 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.520100    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"
Nov 03 14:53:36.520351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.520126    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td" event={"ID":"226c6820-71f4-4b03-92a7-2c524a77e9da","Type":"ContainerDied","Data":"bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809"}
Nov 03 14:53:36.520351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.520165    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td" event={"ID":"226c6820-71f4-4b03-92a7-2c524a77e9da","Type":"ContainerDied","Data":"e97e08baadd836a9b4548c7522aeb94509fc0879881159a7bbc8ecda4c04299f"}
Nov 03 14:53:36.520351 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.520183    6481 scope.go:122] "RemoveContainer" containerID="bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809"
Nov 03 14:53:36.529531 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.529516    6481 scope.go:122] "RemoveContainer" containerID="bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809"
Nov 03 14:53:36.529823 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:53:36.529806    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809\": container with ID starting with bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809 not found: ID does not exist" containerID="bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809"
Nov 03 14:53:36.529877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.529833    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809"} err="failed to get container status \"bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809\": rpc error: code = NotFound desc = could not find container \"bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809\": container with ID starting with bfde9442b51a0c66cb2492417759818da14b5a0ab27927f8b3ca221b036c6809 not found: ID does not exist"
Nov 03 14:53:36.573853 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.573828    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"]
Nov 03 14:53:36.585395 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:36.585367    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0e21d4af-kserve-7cd98c55f7-wx4td"]
Nov 03 14:53:37.039693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:37.039655    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"]
Nov 03 14:53:37.051443 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:37.051415    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4bp8"]
Nov 03 14:53:38.065102 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:38.065072    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226c6820-71f4-4b03-92a7-2c524a77e9da" path="/var/lib/kubelet/pods/226c6820-71f4-4b03-92a7-2c524a77e9da/volumes"
Nov 03 14:53:38.065510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:38.065495    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9105326e-6185-47f7-b4e0-dd4f1f61f9cf" path="/var/lib/kubelet/pods/9105326e-6185-47f7-b4e0-dd4f1f61f9cf/volumes"
Nov 03 14:53:42.351507 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.351471    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:53:42.499499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.499434    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r" podUID="4c323af7-589b-414c-9296-e644f08505ee" containerName="main" containerID="cri-o://a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261" gracePeriod=45
Nov 03 14:53:42.849263 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.849060    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:53:42.989484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.989447    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir\") pod \"4c323af7-589b-414c-9296-e644f08505ee\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") "
Nov 03 14:53:42.989708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.989520    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n\" (UniqueName: \"kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n\") pod \"4c323af7-589b-414c-9296-e644f08505ee\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") "
Nov 03 14:53:42.989708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.989556    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm\") pod \"4c323af7-589b-414c-9296-e644f08505ee\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") "
Nov 03 14:53:42.989708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.989596    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home\") pod \"4c323af7-589b-414c-9296-e644f08505ee\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") "
Nov 03 14:53:42.989708 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.989673    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs\") pod \"4c323af7-589b-414c-9296-e644f08505ee\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") "
Nov 03 14:53:42.989976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.989712    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache\") pod \"4c323af7-589b-414c-9296-e644f08505ee\" (UID: \"4c323af7-589b-414c-9296-e644f08505ee\") "
Nov 03 14:53:42.994262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.990335    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache" pod "4c323af7-589b-414c-9296-e644f08505ee" (UID: "4c323af7-589b-414c-9296-e644f08505ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:42.994262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.990543    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir" pod "4c323af7-589b-414c-9296-e644f08505ee" (UID: "4c323af7-589b-414c-9296-e644f08505ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:42.994262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.991050    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home" pod "4c323af7-589b-414c-9296-e644f08505ee" (UID: "4c323af7-589b-414c-9296-e644f08505ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:42.994262 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.993221    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n" pod "4c323af7-589b-414c-9296-e644f08505ee" (UID: "4c323af7-589b-414c-9296-e644f08505ee"). InnerVolumeSpecName "kube-api-access-kvw5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:53:42.995552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:42.995496    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs" pod "4c323af7-589b-414c-9296-e644f08505ee" (UID: "4c323af7-589b-414c-9296-e644f08505ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:53:43.054816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.054759    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm" pod "4c323af7-589b-414c-9296-e644f08505ee" (UID: "4c323af7-589b-414c-9296-e644f08505ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:53:43.100559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.098904    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c323af7-589b-414c-9296-e644f08505ee-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:43.100559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.098946    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:43.100559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.098962    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:43.100559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.098976    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvw5n\" (UniqueName: \"kubernetes.io/projected/4c323af7-589b-414c-9296-e644f08505ee-kube-api-access-kvw5n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:43.100559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.098993    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:43.100559 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.099006    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c323af7-589b-414c-9296-e644f08505ee-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:43.565292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.564782    6481 generic.go:356] "Generic (PLEG): container finished" podID="4c323af7-589b-414c-9296-e644f08505ee" containerID="a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261" exitCode=0
Nov 03 14:53:43.565292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.564871    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r" event={"ID":"4c323af7-589b-414c-9296-e644f08505ee","Type":"ContainerDied","Data":"a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261"}
Nov 03 14:53:43.565292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.564909    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r" event={"ID":"4c323af7-589b-414c-9296-e644f08505ee","Type":"ContainerDied","Data":"589ae097f4a08089aab548f9e0148746587e6955bd803cd9d2da7b4a0f912dc4"}
Nov 03 14:53:43.565292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.564935    6481 scope.go:122] "RemoveContainer" containerID="a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261"
Nov 03 14:53:43.565292 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.565085    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"
Nov 03 14:53:43.618605 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.618500    6481 scope.go:122] "RemoveContainer" containerID="a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261"
Nov 03 14:53:43.622670 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:53:43.622568    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261\": container with ID starting with a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261 not found: ID does not exist" containerID="a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261"
Nov 03 14:53:43.622670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.622609    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261"} err="failed to get container status \"a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261\": rpc error: code = NotFound desc = could not find container \"a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261\": container with ID starting with a9049459132e64722e2847c0554f9428c595ab8b1cc1834d603f9cd5400e7261 not found: ID does not exist"
Nov 03 14:53:43.655653 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.655625    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"]
Nov 03 14:53:43.668515 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:43.668453    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-0e21d4af-kserve-6b856bff8f-phj2r"]
Nov 03 14:53:44.080254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:44.079824    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c323af7-589b-414c-9296-e644f08505ee" path="/var/lib/kubelet/pods/4c323af7-589b-414c-9296-e644f08505ee/volumes"
Nov 03 14:53:46.997593 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:46.997553    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:53:48.040076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:48.040039    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"]
Nov 03 14:53:48.051451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:48.051421    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hb4hw"]
Nov 03 14:53:48.069512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:48.069473    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf61615-a11d-4683-8494-8bbd50b00d7e" path="/var/lib/kubelet/pods/3bf61615-a11d-4683-8494-8bbd50b00d7e/volumes"
Nov 03 14:53:52.443635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.443599    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"]
Nov 03 14:53:52.444031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444013    6481 cpu_manager.go:414] "containerMap: removing container" podUID="226c6820-71f4-4b03-92a7-2c524a77e9da" containerName="main"
Nov 03 14:53:52.444031 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444025    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226c6820-71f4-4b03-92a7-2c524a77e9da" containerName="main"
Nov 03 14:53:52.444110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444049    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5882cd8d-1423-434d-9538-5e35ad117da9" containerName="revoke-keys"
Nov 03 14:53:52.444110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444057    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5882cd8d-1423-434d-9538-5e35ad117da9" containerName="revoke-keys"
Nov 03 14:53:52.444110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444075    6481 cpu_manager.go:414] "containerMap: removing container" podUID="4c323af7-589b-414c-9296-e644f08505ee" containerName="main"
Nov 03 14:53:52.444110 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444081    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c323af7-589b-414c-9296-e644f08505ee" containerName="main"
Nov 03 14:53:52.444268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444170    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c323af7-589b-414c-9296-e644f08505ee" containerName="main"
Nov 03 14:53:52.444268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444183    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="226c6820-71f4-4b03-92a7-2c524a77e9da" containerName="main"
Nov 03 14:53:52.444268 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.444191    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5882cd8d-1423-434d-9538-5e35ad117da9" containerName="revoke-keys"
Nov 03 14:53:52.449076 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.449050    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.467542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.467518    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"]
Nov 03 14:53:52.476597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.476579    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:53:52.476887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.476864    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:53:52.477016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.476956    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:53:52.477016 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.477002    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:53:52.559867 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.559830    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.560019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.559947    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7vl\" (UniqueName: \"kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.560019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.559991    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.661444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.661410    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7vl\" (UniqueName: \"kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.661638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.661453    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.661638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.661518    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.662247 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.662202    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.663801 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.663783    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.683124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.683097    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7vl\" (UniqueName: \"kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl\") pod \"maas-api-revoke-sub-cafa65a1-b4ngc\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:52.761951 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:52.761899    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:53.059355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:53.059330    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"]
Nov 03 14:53:53.061640 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:53:53.061606    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdee3c26_dea8_47ff_a826_38ab91455003.slice/crio-d904d5603b609132819cae5aaef92dfeb88e069e0fde4434d89b0e91fa390bd9 WatchSource:0}: Error finding container d904d5603b609132819cae5aaef92dfeb88e069e0fde4434d89b0e91fa390bd9: Status 404 returned error can't find the container with id d904d5603b609132819cae5aaef92dfeb88e069e0fde4434d89b0e91fa390bd9
Nov 03 14:53:53.689622 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:53.689588    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc" event={"ID":"bdee3c26-dea8-47ff-a826-38ab91455003","Type":"ContainerStarted","Data":"c9450411f21c77325500a60bc5011b76e20b6a4f6ea1932e7bdcce34d8980117"}
Nov 03 14:53:53.690090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:53.689644    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc" event={"ID":"bdee3c26-dea8-47ff-a826-38ab91455003","Type":"ContainerStarted","Data":"d904d5603b609132819cae5aaef92dfeb88e069e0fde4434d89b0e91fa390bd9"}
Nov 03 14:53:53.730664 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:53.730616    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc" podStartSLOduration=1.730602646 podStartE2EDuration="1.730602646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:53:53.729397282 +0000 UTC m=+2511.779428464" watchObservedRunningTime="2026-11-03 14:53:53.730602646 +0000 UTC m=+2511.780633878"
Nov 03 14:53:54.696525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:54.696492    6481 generic.go:356] "Generic (PLEG): container finished" podID="bdee3c26-dea8-47ff-a826-38ab91455003" containerID="c9450411f21c77325500a60bc5011b76e20b6a4f6ea1932e7bdcce34d8980117" exitCode=0
Nov 03 14:53:54.696958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:54.696582    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc" event={"ID":"bdee3c26-dea8-47ff-a826-38ab91455003","Type":"ContainerDied","Data":"c9450411f21c77325500a60bc5011b76e20b6a4f6ea1932e7bdcce34d8980117"}
Nov 03 14:53:55.837403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.837377    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:55.893078 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.893050    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl\" (UniqueName: \"kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl\") pod \"bdee3c26-dea8-47ff-a826-38ab91455003\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") "
Nov 03 14:53:55.893214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.893086    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca\") pod \"bdee3c26-dea8-47ff-a826-38ab91455003\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") "
Nov 03 14:53:55.893214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.893154    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca\") pod \"bdee3c26-dea8-47ff-a826-38ab91455003\" (UID: \"bdee3c26-dea8-47ff-a826-38ab91455003\") "
Nov 03 14:53:55.893583 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.893560    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca" pod "bdee3c26-dea8-47ff-a826-38ab91455003" (UID: "bdee3c26-dea8-47ff-a826-38ab91455003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:53:55.895245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.895201    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca" pod "bdee3c26-dea8-47ff-a826-38ab91455003" (UID: "bdee3c26-dea8-47ff-a826-38ab91455003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:53:55.895338 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.895299    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl" pod "bdee3c26-dea8-47ff-a826-38ab91455003" (UID: "bdee3c26-dea8-47ff-a826-38ab91455003"). InnerVolumeSpecName "kube-api-access-jf7vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:53:55.994484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.994420    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7vl\" (UniqueName: \"kubernetes.io/projected/bdee3c26-dea8-47ff-a826-38ab91455003-kube-api-access-jf7vl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:55.994484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.994443    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:55.994484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:55.994453    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdee3c26-dea8-47ff-a826-38ab91455003-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:53:56.714572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:56.714545    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"
Nov 03 14:53:56.714572 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:56.714566    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc" event={"ID":"bdee3c26-dea8-47ff-a826-38ab91455003","Type":"ContainerDied","Data":"d904d5603b609132819cae5aaef92dfeb88e069e0fde4434d89b0e91fa390bd9"}
Nov 03 14:53:56.714789 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:53:56.714609    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d904d5603b609132819cae5aaef92dfeb88e069e0fde4434d89b0e91fa390bd9"
Nov 03 14:53:57.986557 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:53:57.986517    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:54:01.409086 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:01.409047    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:03.883172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:03.883141    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:54:07.827004 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:07.826973    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:08.163512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.163463    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"]
Nov 03 14:54:08.164063 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.164039    6481 cpu_manager.go:414] "containerMap: removing container" podUID="bdee3c26-dea8-47ff-a826-38ab91455003" containerName="revoke-keys"
Nov 03 14:54:08.164271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.164242    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdee3c26-dea8-47ff-a826-38ab91455003" containerName="revoke-keys"
Nov 03 14:54:08.164403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.164388    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdee3c26-dea8-47ff-a826-38ab91455003" containerName="revoke-keys"
Nov 03 14:54:08.167705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.167687    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.183626 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.183605    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-72b663-openshift-default-dockercfg-4v4k6\""
Nov 03 14:54:08.196860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.196831    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"]
Nov 03 14:54:08.309654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309283    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309361    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqbr\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309526    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309619    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309676    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309730    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309782    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309832    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.309998 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.309861    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.317025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.316991    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:08.410561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410530    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.410561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410566    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.410775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410592    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.410775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410633    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.410775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410666    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.410775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410711    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.410775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410736    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410800    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.410830    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqbr\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.411046    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.411088    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.411115    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.411158    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.411552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.411536    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.412840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.412820    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.413210 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.413195    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.426364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.426313    6481 scope.go:122] "RemoveContainer" containerID="f2f1e6ececad53505a39a70bc0ecd1d3a4b577f08e2803091cea4c62ebd5824e"
Nov 03 14:54:08.433255 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.433219    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqbr\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.434667 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.434647    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token\") pod \"e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") " pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.483245 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.483196    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:08.520175 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.520140    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:08.802601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.800500    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"]
Nov 03 14:54:08.805331 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.805271    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:08.805475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.805412    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:08.805539 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:08.805492    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:09.784184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:09.784145    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" event={"ID":"5c748297-e5da-49fa-ac16-998070f730bc","Type":"ContainerStarted","Data":"84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"}
Nov 03 14:54:09.784691 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:09.784196    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" event={"ID":"5c748297-e5da-49fa-ac16-998070f730bc","Type":"ContainerStarted","Data":"f6a96ceb57f785ef49da57abe7bb63f3f8e178f5c5acbb7b115fcbc707fc1ee0"}
Nov 03 14:54:09.852066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:09.852019    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podStartSLOduration=1.852006099 podStartE2EDuration="1.852006099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:09.839317638 +0000 UTC m=+2527.889348821" watchObservedRunningTime="2026-11-03 14:54:09.852006099 +0000 UTC m=+2527.902037261"
Nov 03 14:54:10.483606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:10.483574    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:10.488378 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:10.488351    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:10.789571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:10.789487    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:10.790452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:10.790433    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:12.417017 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:12.416990    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:12.632842 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:12.632804    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:14.033883 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.033784    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:14.241875 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.241834    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"]
Nov 03 14:54:14.245656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.245635    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.250196 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.250173    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:14.266575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.266553    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"]
Nov 03 14:54:14.273146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.273116    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-72b663\""
Nov 03 14:54:14.273146 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.273135    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-72b663-dockercfg-5dnqq\""
Nov 03 14:54:14.350357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.350264    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"]
Nov 03 14:54:14.354681 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.354654    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.372717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.372685    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"]
Nov 03 14:54:14.374207 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.374182    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.374375 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.374356    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjv2n\" (UniqueName: \"kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n\") pod \"payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.475483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.475446    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.475697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.475509    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbj8\" (UniqueName: \"kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8\") pod \"payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.475697 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.475643    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjv2n\" (UniqueName: \"kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n\") pod \"payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.475806 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.475707    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.476149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.476131    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.496516 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.496491    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjv2n\" (UniqueName: \"kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n\") pod \"payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.559280 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.559257    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:14.577003 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.576974    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.577104 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.577068    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbj8\" (UniqueName: \"kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8\") pod \"payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.577666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.577647    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.599254 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.597983    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbj8\" (UniqueName: \"kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8\") pod \"payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.669474 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.669447    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:14.858783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.858759    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"]
Nov 03 14:54:14.860289 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:54:14.860261    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdd0e53_feae_4679_9978_9413eeb5229b.slice/crio-cd848f4114e11636b4863f3ba35308469c9831ce01c97ad232c0aa433f39a17f WatchSource:0}: Error finding container cd848f4114e11636b4863f3ba35308469c9831ce01c97ad232c0aa433f39a17f: Status 404 returned error can't find the container with id cd848f4114e11636b4863f3ba35308469c9831ce01c97ad232c0aa433f39a17f
Nov 03 14:54:14.970339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:14.970306    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"]
Nov 03 14:54:15.816311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.816272    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" event={"ID":"0cdd0e53-feae-4679-9978-9413eeb5229b","Type":"ContainerStarted","Data":"6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8"}
Nov 03 14:54:15.816775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.816331    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" event={"ID":"0cdd0e53-feae-4679-9978-9413eeb5229b","Type":"ContainerStarted","Data":"cd848f4114e11636b4863f3ba35308469c9831ce01c97ad232c0aa433f39a17f"}
Nov 03 14:54:15.816775 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.816448    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:15.817919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.817895    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" event={"ID":"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659","Type":"ContainerStarted","Data":"a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca"}
Nov 03 14:54:15.818040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.817934    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" event={"ID":"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659","Type":"ContainerStarted","Data":"6760ee578e6497f41d4c30d01b5d77fb454eec09877473d8f04b6163717a86bc"}
Nov 03 14:54:15.818040 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.817985    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:15.858153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.858117    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" podStartSLOduration=1.8581051610000001 podStartE2EDuration="1.858105161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:15.85705458 +0000 UTC m=+2533.907085763" watchObservedRunningTime="2026-11-03 14:54:15.858105161 +0000 UTC m=+2533.908136344"
Nov 03 14:54:15.901171 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.901129    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" podStartSLOduration=1.90111835 podStartE2EDuration="1.90111835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:15.899852477 +0000 UTC m=+2533.949883660" watchObservedRunningTime="2026-11-03 14:54:15.90111835 +0000 UTC m=+2533.951149534"
Nov 03 14:54:15.981350 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:15.981322    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:16.782411 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:16.782379    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"]
Nov 03 14:54:16.819152 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:16.819125    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"]
Nov 03 14:54:17.000676 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.000641    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.000879 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.000730    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume podName:0cdd0e53-feae-4679-9978-9413eeb5229b nodeName:}" failed. No retries permitted until 2026-11-03 14:54:17.500710698 +0000 UTC m=+2535.550741886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" (UID: "0cdd0e53-feae-4679-9978-9413eeb5229b") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.000946 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.000910    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.001007 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.000994    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume podName:5261f8f1-64ea-4f8b-bd5d-a5f5bd895659 nodeName:}" failed. No retries permitted until 2026-11-03 14:54:17.500974928 +0000 UTC m=+2535.551006114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume") pod "payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" (UID: "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.036380 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:17.036279    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:17.242557 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:17.242526    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:17.506682 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.506645    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.506879 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.506713    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.506879 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.506731    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume podName:0cdd0e53-feae-4679-9978-9413eeb5229b nodeName:}" failed. No retries permitted until 2026-11-03 14:54:18.506712521 +0000 UTC m=+2536.556743697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" (UID: "0cdd0e53-feae-4679-9978-9413eeb5229b") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.506879 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:17.506782    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume podName:5261f8f1-64ea-4f8b-bd5d-a5f5bd895659 nodeName:}" failed. No retries permitted until 2026-11-03 14:54:18.506765251 +0000 UTC m=+2536.556796428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume") pod "payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" (UID: "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:17.869475 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:17.869380    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:54:18.518857 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:18.518824    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:18.519037 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:18.518867    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:18.519037 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:18.518910    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume podName:0cdd0e53-feae-4679-9978-9413eeb5229b nodeName:}" failed. No retries permitted until 2026-11-03 14:54:20.518892411 +0000 UTC m=+2538.568923585 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" (UID: "0cdd0e53-feae-4679-9978-9413eeb5229b") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:18.519037 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:18.518928    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume podName:5261f8f1-64ea-4f8b-bd5d-a5f5bd895659 nodeName:}" failed. No retries permitted until 2026-11-03 14:54:20.518920175 +0000 UTC m=+2538.568951351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume") pod "payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" (UID: "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:19.834072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:19.834039    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:20.546273 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:20.546216    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:20.546488 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:20.546316    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume podName:0cdd0e53-feae-4679-9978-9413eeb5229b nodeName:}" failed. No retries permitted until 2026-11-03 14:54:24.546299883 +0000 UTC m=+2542.596331044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume") pod "payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" (UID: "0cdd0e53-feae-4679-9978-9413eeb5229b") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:20.546488 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:20.546337    6481 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-ipp-a-72b663: configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:20.546488 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:20.546394    6481 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume podName:5261f8f1-64ea-4f8b-bd5d-a5f5bd895659 nodeName:}" failed. No retries permitted until 2026-11-03 14:54:24.546381539 +0000 UTC m=+2542.596412705 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume") pod "payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" (UID: "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659") : configmap "payload-processing-plugins-e2e-ipp-a-72b663" not found
Nov 03 14:54:21.192185 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.192128    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:21.413246 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.413199    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:21.673495 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.673462    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"]
Nov 03 14:54:21.677311 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.677294    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.698065 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.698044    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"]
Nov 03 14:54:21.727321 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.710506    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-72b663-dockercfg-nfkm9\""
Nov 03 14:54:21.761295 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.761267    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.761430 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.761304    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmcj\" (UniqueName: \"kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj\") pod \"payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.862277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.862212    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmcj\" (UniqueName: \"kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj\") pod \"payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.862488 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.862473    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.863134 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.863110    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.883325 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.883299    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmcj\" (UniqueName: \"kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj\") pod \"payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:21.990400 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:21.990326    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:22.331722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.331289    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"]
Nov 03 14:54:22.334950 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:54:22.334914    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31e94e1_7ccd_49e0_a98b_2d96c6f77de4.slice/crio-f242a07d5c5c99de87d8740551893ca8b33eba57141d5e410d6e17e3747719a3 WatchSource:0}: Error finding container f242a07d5c5c99de87d8740551893ca8b33eba57141d5e410d6e17e3747719a3: Status 404 returned error can't find the container with id f242a07d5c5c99de87d8740551893ca8b33eba57141d5e410d6e17e3747719a3
Nov 03 14:54:22.337436 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.337406    6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 14:54:22.398305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.398274    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:22.769439 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.769404    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:22.827942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.827900    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" podUID="0cdd0e53-feae-4679-9978-9413eeb5229b" containerName="payload-pre-processing" containerID="cri-o://6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8" gracePeriod=25
Nov 03 14:54:22.828140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.827911    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" podUID="5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" containerName="payload-processing" containerID="cri-o://a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca" gracePeriod=25
Nov 03 14:54:22.856937 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:22.856897    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj" event={"ID":"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4","Type":"ContainerStarted","Data":"f242a07d5c5c99de87d8740551893ca8b33eba57141d5e410d6e17e3747719a3"}
Nov 03 14:54:23.150314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.150283    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:23.177851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.177412    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume\") pod \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") "
Nov 03 14:54:23.177851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.177482    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8\" (UniqueName: \"kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8\") pod \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\" (UID: \"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659\") "
Nov 03 14:54:23.177851 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.177809    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume" pod "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" (UID: "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:54:23.178121 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.177951    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:54:23.179848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.179828    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8" pod "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" (UID: "5261f8f1-64ea-4f8b-bd5d-a5f5bd895659"). InnerVolumeSpecName "kube-api-access-5zbj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:54:23.185000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.184977    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:23.278846 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.278767    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume\") pod \"0cdd0e53-feae-4679-9978-9413eeb5229b\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") "
Nov 03 14:54:23.279010 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.278910    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n\" (UniqueName: \"kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n\") pod \"0cdd0e53-feae-4679-9978-9413eeb5229b\" (UID: \"0cdd0e53-feae-4679-9978-9413eeb5229b\") "
Nov 03 14:54:23.279202 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.279182    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume" pod "0cdd0e53-feae-4679-9978-9413eeb5229b" (UID: "0cdd0e53-feae-4679-9978-9413eeb5229b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:54:23.279327 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.279309    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cdd0e53-feae-4679-9978-9413eeb5229b-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:54:23.279364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.279334    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zbj8\" (UniqueName: \"kubernetes.io/projected/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659-kube-api-access-5zbj8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:54:23.281417 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.281391    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n" pod "0cdd0e53-feae-4679-9978-9413eeb5229b" (UID: "0cdd0e53-feae-4679-9978-9413eeb5229b"). InnerVolumeSpecName "kube-api-access-wjv2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:54:23.380705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.380676    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjv2n\" (UniqueName: \"kubernetes.io/projected/0cdd0e53-feae-4679-9978-9413eeb5229b-kube-api-access-wjv2n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:54:23.863594 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.863554    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj" event={"ID":"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4","Type":"ContainerStarted","Data":"68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363"}
Nov 03 14:54:23.863797 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.863710    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:23.864931 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.864903    6481 generic.go:356] "Generic (PLEG): container finished" podID="0cdd0e53-feae-4679-9978-9413eeb5229b" containerID="6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8" exitCode=0
Nov 03 14:54:23.865074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.864939    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"
Nov 03 14:54:23.865074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.864978    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" event={"ID":"0cdd0e53-feae-4679-9978-9413eeb5229b","Type":"ContainerDied","Data":"6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8"}
Nov 03 14:54:23.865074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.865016    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92" event={"ID":"0cdd0e53-feae-4679-9978-9413eeb5229b","Type":"ContainerDied","Data":"cd848f4114e11636b4863f3ba35308469c9831ce01c97ad232c0aa433f39a17f"}
Nov 03 14:54:23.865074 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.865039    6481 scope.go:122] "RemoveContainer" containerID="6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8"
Nov 03 14:54:23.866346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.866328    6481 generic.go:356] "Generic (PLEG): container finished" podID="5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" containerID="a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca" exitCode=0
Nov 03 14:54:23.866457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.866356    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" event={"ID":"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659","Type":"ContainerDied","Data":"a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca"}
Nov 03 14:54:23.866457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.866378    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"
Nov 03 14:54:23.866457 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.866411    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p" event={"ID":"5261f8f1-64ea-4f8b-bd5d-a5f5bd895659","Type":"ContainerDied","Data":"6760ee578e6497f41d4c30d01b5d77fb454eec09877473d8f04b6163717a86bc"}
Nov 03 14:54:23.875587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.875571    6481 scope.go:122] "RemoveContainer" containerID="6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8"
Nov 03 14:54:23.875869 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:23.875849    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8\": container with ID starting with 6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8 not found: ID does not exist" containerID="6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8"
Nov 03 14:54:23.875925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.875877    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8"} err="failed to get container status \"6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8\": rpc error: code = NotFound desc = could not find container \"6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8\": container with ID starting with 6629530f384bbcceba86a866d367f47e903f72e4d46230b73e2673526b9b14a8 not found: ID does not exist"
Nov 03 14:54:23.875925 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.875895    6481 scope.go:122] "RemoveContainer" containerID="a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca"
Nov 03 14:54:23.884885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.884869    6481 scope.go:122] "RemoveContainer" containerID="a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca"
Nov 03 14:54:23.885141 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:23.885121    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca\": container with ID starting with a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca not found: ID does not exist" containerID="a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca"
Nov 03 14:54:23.885211 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.885150    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca"} err="failed to get container status \"a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca\": rpc error: code = NotFound desc = could not find container \"a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca\": container with ID starting with a15e43832be413fcb94c9871029f8131452e997d70db7f1ccb10dcbf9916cfca not found: ID does not exist"
Nov 03 14:54:23.920260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.918049    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj" podStartSLOduration=1.559081701 podStartE2EDuration="2.91802972s" totalImagesPullingTime="1.358948019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:23.91663249 +0000 UTC m=+2541.966663676" watchObservedRunningTime="2026-11-03 14:54:23.91802972 +0000 UTC m=+2541.968060903"
Nov 03 14:54:23.955468 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.955399    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"]
Nov 03 14:54:23.963363 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.963339    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-72b663-656bdbd5ff-slc92"]
Nov 03 14:54:23.997464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:23.997438    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"]
Nov 03 14:54:24.003717 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:24.003688    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-596b566c8f-vvw4p"]
Nov 03 14:54:24.067663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:24.067630    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdd0e53-feae-4679-9978-9413eeb5229b" path="/var/lib/kubelet/pods/0cdd0e53-feae-4679-9978-9413eeb5229b/volumes"
Nov 03 14:54:24.068024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:24.068012    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" path="/var/lib/kubelet/pods/5261f8f1-64ea-4f8b-bd5d-a5f5bd895659/volumes"
Nov 03 14:54:24.302716 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:24.302628    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:26.681445 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:26.681406    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:26.831428 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:26.831388    6481 patch_prober.go:29] interesting pod/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 14:54:26.831582 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:26.831462    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 14:54:27.655291 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.655194    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:27.891746 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.891713    6481 generic.go:356] "Generic (PLEG): container finished" podID="5c748297-e5da-49fa-ac16-998070f730bc" containerID="84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947" exitCode=0
Nov 03 14:54:27.892166 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.891792    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" event={"ID":"5c748297-e5da-49fa-ac16-998070f730bc","Type":"ContainerDied","Data":"84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"}
Nov 03 14:54:27.892220 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.892183    6481 scope.go:122] "RemoveContainer" containerID="84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"
Nov 03 14:54:27.893396 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.893366    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:27.893464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.893438    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:27.893511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:27.893488    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:27.996080 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:27.996038    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:54:28.726163 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:28.726050    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:28.898406 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:28.898372    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" event={"ID":"5c748297-e5da-49fa-ac16-998070f730bc","Type":"ContainerStarted","Data":"bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709"}
Nov 03 14:54:29.483929 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:29.483891    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:29.484149 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:29.484124    6481 patch_prober.go:29] interesting pod/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 14:54:29.484257 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:29.484183    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 14:54:30.484581 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:30.484538    6481 patch_prober.go:29] interesting pod/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 14:54:30.484987 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:30.484605    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 14:54:31.158638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:31.158601    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:31.483848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:31.483757    6481 patch_prober.go:29] interesting pod/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 14:54:31.483848 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:31.483812    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 14:54:32.484124 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:32.484092    6481 patch_prober.go:29] interesting pod/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 14:54:32.484565 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:32.484153    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 14:54:33.113742 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:33.113706    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:33.484599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:33.484571    6481 patch_prober.go:29] interesting pod/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Nov 03 14:54:33.484991 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:33.484631    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Nov 03 14:54:34.487576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:34.487548    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:34.488106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:34.487803    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:34.488471 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:34.488453    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:54:34.876781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:34.876707    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:54:35.485693 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:35.485662    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:37.412394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:37.412359    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:38.011497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:38.011464    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:38.220170 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:38.220139    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:38.906106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:38.906079    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:39.204177 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204142    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"]
Nov 03 14:54:39.204650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204632    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0cdd0e53-feae-4679-9978-9413eeb5229b" containerName="payload-pre-processing"
Nov 03 14:54:39.204751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204652    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdd0e53-feae-4679-9978-9413eeb5229b" containerName="payload-pre-processing"
Nov 03 14:54:39.204751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204702    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" containerName="payload-processing"
Nov 03 14:54:39.204751 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204711    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" containerName="payload-processing"
Nov 03 14:54:39.204919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204824    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5261f8f1-64ea-4f8b-bd5d-a5f5bd895659" containerName="payload-processing"
Nov 03 14:54:39.204919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.204845    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdd0e53-feae-4679-9978-9413eeb5229b" containerName="payload-pre-processing"
Nov 03 14:54:39.210265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.210219    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.231493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.231467    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"]
Nov 03 14:54:39.239942 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.239921    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-131649-openshift-default-dockercfg-wxj7d\""
Nov 03 14:54:39.349365 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349337    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349372    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349391    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349512 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349409    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349534    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349559    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349585    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349693    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.349805 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.349755    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5g2c\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.352025 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.352006    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:39.451106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451070    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451106 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451111    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451136    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451164    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451408 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451197    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5g2c\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451439    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451493    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451520    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451550    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451589 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451579    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.451969 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.451943    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.452103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.452044    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.452103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.452083    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.452312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.452296    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.453635 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.453613    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.453816 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.453799    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.471779 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.471726    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.471957 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.471933    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5g2c\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c\") pod \"e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") " pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.525172 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.525143    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:39.558496 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.558456    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:39.837419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.837334    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:39.840276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.840252    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"]
Nov 03 14:54:39.842216 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:54:39.842192    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc009545a_8a66_4f40_8407_d958bd1df4c6.slice/crio-2f109f2eedcc250aa89df9828ea2661722f63be9118741598465b43a7b968b6f WatchSource:0}: Error finding container 2f109f2eedcc250aa89df9828ea2661722f63be9118741598465b43a7b968b6f: Status 404 returned error can't find the container with id 2f109f2eedcc250aa89df9828ea2661722f63be9118741598465b43a7b968b6f
Nov 03 14:54:39.844510 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.844478    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:39.844597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.844544    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:39.844597 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.844592    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:54:39.980832 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.980796    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" event={"ID":"c009545a-8a66-4f40-8407-d958bd1df4c6","Type":"ContainerStarted","Data":"ebfd7ab4ba1b109eea487a3eff92f14cab33b7a4f2e6964e97c1f15e8ded0417"}
Nov 03 14:54:39.981219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:39.980859    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" event={"ID":"c009545a-8a66-4f40-8407-d958bd1df4c6","Type":"ContainerStarted","Data":"2f109f2eedcc250aa89df9828ea2661722f63be9118741598465b43a7b968b6f"}
Nov 03 14:54:40.026082 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:40.026027    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podStartSLOduration=1.026013166 podStartE2EDuration="1.026013166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:40.025104874 +0000 UTC m=+2558.075136069" watchObservedRunningTime="2026-11-03 14:54:40.026013166 +0000 UTC m=+2558.076044350"
Nov 03 14:54:40.525463 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:40.525431    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:40.530464 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:40.530438    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:40.986458 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:40.986426    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:40.987343 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:40.987325    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:54:41.862297 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:41.862263    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:42.519756 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:42.519725    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:42.727764 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:42.727732    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:44.295269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:44.295241    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:44.506628 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:44.506599    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:46.287948 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:46.287914    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:46.505705 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:46.505667    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:46.711690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:46.711656    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:47.088414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:47.088339    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:47.297360 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:47.297313    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:48.629774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:48.629745    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:50.168433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:50.168396    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:54:50.675043 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:50.675008    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:52.639131 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:52.639100    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:54.028281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.028249    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:54.233346 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.233310    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:54.333294 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.333191    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"]
Nov 03 14:54:54.337021 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.337001    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.363155 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.363118    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-131649-dockercfg-85jmk\""
Nov 03 14:54:54.363328 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.363252    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-131649\""
Nov 03 14:54:54.365374 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.365348    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"]
Nov 03 14:54:54.441654 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.441620    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.441840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.441668    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j8t\" (UniqueName: \"kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t\") pod \"payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.542682 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.542652    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.542857 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.542695    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j8t\" (UniqueName: \"kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t\") pod \"payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.543425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.543403    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.565707 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.565674    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j8t\" (UniqueName: \"kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t\") pod \"payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.650566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.650498    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:54.978277 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:54.978249    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"]
Nov 03 14:54:54.979812 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:54:54.979784    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2741c5f4_d9b5_442a_bcee_44cf3a0899c4.slice/crio-4b7cec3af8593efb300febe705101801d1473c28bd61abdbb59198466d537c45 WatchSource:0}: Error finding container 4b7cec3af8593efb300febe705101801d1473c28bd61abdbb59198466d537c45: Status 404 returned error can't find the container with id 4b7cec3af8593efb300febe705101801d1473c28bd61abdbb59198466d537c45
Nov 03 14:54:55.073158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:55.073125    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl" event={"ID":"2741c5f4-d9b5-442a-bcee-44cf3a0899c4","Type":"ContainerStarted","Data":"cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08"}
Nov 03 14:54:55.073552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:55.073190    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl" event={"ID":"2741c5f4-d9b5-442a-bcee-44cf3a0899c4","Type":"ContainerStarted","Data":"4b7cec3af8593efb300febe705101801d1473c28bd61abdbb59198466d537c45"}
Nov 03 14:54:55.073552 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:55.073302    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:54:55.326324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:55.326219    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:57.666868 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:57.666829    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:54:57.986512 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:54:57.986429    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:54:59.613887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:54:59.613834    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:00.100638 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.100611    6481 generic.go:356] "Generic (PLEG): container finished" podID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerID="ebfd7ab4ba1b109eea487a3eff92f14cab33b7a4f2e6964e97c1f15e8ded0417" exitCode=0
Nov 03 14:55:00.100959 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.100686    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" event={"ID":"c009545a-8a66-4f40-8407-d958bd1df4c6","Type":"ContainerDied","Data":"ebfd7ab4ba1b109eea487a3eff92f14cab33b7a4f2e6964e97c1f15e8ded0417"}
Nov 03 14:55:00.101046 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.100991    6481 scope.go:122] "RemoveContainer" containerID="ebfd7ab4ba1b109eea487a3eff92f14cab33b7a4f2e6964e97c1f15e8ded0417"
Nov 03 14:55:00.102392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.102358    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:55:00.102518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.102429    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:55:00.102518 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.102461    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 14:55:00.167612 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:00.167568    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl" podStartSLOduration=6.167552703 podStartE2EDuration="6.167552703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:54:55.122515857 +0000 UTC m=+2573.172547041" watchObservedRunningTime="2026-11-03 14:55:00.167552703 +0000 UTC m=+2578.217583886"
Nov 03 14:55:01.130726 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:01.130693    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" event={"ID":"c009545a-8a66-4f40-8407-d958bd1df4c6","Type":"ContainerStarted","Data":"db61207fe86b9285b41475729495c09011da8f849f81c53412057dc877c49e12"}
Nov 03 14:55:01.525885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:01.525839    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:55:01.526160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:01.526061    6481 patch_prober.go:29] interesting pod/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 14:55:01.526160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:01.526117    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 14:55:01.526587 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:01.526562    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:02.008926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:02.008748    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:02.525609 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:02.525573    6481 patch_prober.go:29] interesting pod/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 14:55:02.526042 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:02.525638    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 14:55:02.922997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:02.922930    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:03.525641 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:03.525602    6481 patch_prober.go:29] interesting pod/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 14:55:03.526051 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:03.525672    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 14:55:04.482850 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:04.482817    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:04.525561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:04.525526    6481 patch_prober.go:29] interesting pod/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 14:55:04.525858 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:04.525828    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 14:55:05.525580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:05.525538    6481 patch_prober.go:29] interesting pod/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body=
Nov 03 14:55:05.525773 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:05.525606    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused"
Nov 03 14:55:06.080007 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:06.079976    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:55:06.336213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:06.336118    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:06.530281 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:06.530249    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:55:06.530570 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:06.530553    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:55:06.531316 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:06.531296    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:55:08.462364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:08.462328    6481 scope.go:122] "RemoveContainer" containerID="7355b2162ede521290fd85c8f270fc70888ee68a079482d0496f7a712e9cf9a7"
Nov 03 14:55:08.470829 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:08.470808    6481 scope.go:122] "RemoveContainer" containerID="40d213990e9819475b5edac4290b60edce1402c800ef9ee702bca5d5d62f6232"
Nov 03 14:55:08.478663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:08.478642    6481 scope.go:122] "RemoveContainer" containerID="5ca27c54f1430bdfb9dd9ed7588871f0cca36ada323cfdd3b6576e77e96e185b"
Nov 03 14:55:08.906845 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:08.906738    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:11.276135 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:11.276098    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:13.194333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:13.194293    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:15.609503 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:15.609472    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:16.965500 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:16.965471    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:17.995533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:17.995506    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:19.287922 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:19.287889    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:55:19.951284 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:19.951243    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:22.321699 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:22.321665    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:24.677663 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:24.677627    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:26.514511 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:26.514480    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:27.987801 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:55:27.987768    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:55:28.960427 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:28.960396    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:31.296956 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:31.296917    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:33.226561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:33.226516    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:35.630290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:35.630222    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:38.001732 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:38.001687    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:39.451566 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:39.451531    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:39.904383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:39.904293    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:42.296607 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:42.296567    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:44.678856 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:44.678762    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:45.174704 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:45.174672    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:55:46.608993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:46.608942    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:49.012455 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:49.012424    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:51.391953 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:51.391906    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:53.355090 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:53.355057    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:53.855837 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:53.855812    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj_c31e94e1-7ccd-49e0-a98b-2d96c6f77de4/payload-processing/0.log"
Nov 03 14:55:55.651606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:55.651570    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:55.864264 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:55.864213    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:55:57.985054 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:55:57.985009    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:55:58.180364 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:55:58.180328    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:00.117438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:00.117407    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:00.678613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:00.678578    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:01.633088 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:01.633055    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:02.614037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:02.614001    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:04.996324 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:04.996288    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:06.901661 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:06.901634    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:09.304451 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:09.304407    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:11.593590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:11.593558    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:13.602901 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:13.602868    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:16.011012 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:16.010984    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:16.038887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:16.038858    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"]
Nov 03 14:56:16.050355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:16.050329    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea085b5-z4b5f"]
Nov 03 14:56:16.066000 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:16.065962    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="409d059a-c43a-4a3f-b76b-d46ef991eb01" path="/var/lib/kubelet/pods/409d059a-c43a-4a3f-b76b-d46ef991eb01/volumes"
Nov 03 14:56:18.375894 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:18.375862    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:20.301483 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:20.301448    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:22.724952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:22.724919    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:25.186167 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:25.186128    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:27.105971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:27.105937    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:27.985571 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:56:27.985533    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:56:28.267103 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.267025    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"]
Nov 03 14:56:28.270879 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.270858    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.291405 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.291335    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"]
Nov 03 14:56:28.300703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.300678    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:56:28.300808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.300699    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:56:28.300808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.300678    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:56:28.300808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.300786    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:56:28.416293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.416251    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwgjl\" (UniqueName: \"kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.416497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.416329    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.416497 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.416436    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.517371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.517286    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.517371 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.517339    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.517599 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.517466    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwgjl\" (UniqueName: \"kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.518024 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.518004    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.519590 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.519569    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.537818 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.537796    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwgjl\" (UniqueName: \"kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl\") pod \"maas-api-revoke-sub-4d733532-gwtfc\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.582918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.582892    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:28.885290 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:28.885267    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"]
Nov 03 14:56:28.887144 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:56:28.887115    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5a7822_7874_4929_ae68_b1d317a018b4.slice/crio-f818eeeeea72c98967ec0457169be33951c2dde6319971d5a378063a93ba4639 WatchSource:0}: Error finding container f818eeeeea72c98967ec0457169be33951c2dde6319971d5a378063a93ba4639: Status 404 returned error can't find the container with id f818eeeeea72c98967ec0457169be33951c2dde6319971d5a378063a93ba4639
Nov 03 14:56:29.597769 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:29.597682    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:29.608940 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:29.608903    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc" event={"ID":"8c5a7822-7874-4929-ae68-b1d317a018b4","Type":"ContainerStarted","Data":"1f02b0ce6189cd8907a41d130021a435b3401763c018d51364fa691c416f6275"}
Nov 03 14:56:29.609792 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:29.609744    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc" event={"ID":"8c5a7822-7874-4929-ae68-b1d317a018b4","Type":"ContainerStarted","Data":"f818eeeeea72c98967ec0457169be33951c2dde6319971d5a378063a93ba4639"}
Nov 03 14:56:29.654144 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:29.654094    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc" podStartSLOduration=1.654079334 podStartE2EDuration="1.654079334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:56:29.653297784 +0000 UTC m=+2667.703328968" watchObservedRunningTime="2026-11-03 14:56:29.654079334 +0000 UTC m=+2667.704110572"
Nov 03 14:56:30.397841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:30.397806    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:30.616569 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:30.616535    6481 generic.go:356] "Generic (PLEG): container finished" podID="8c5a7822-7874-4929-ae68-b1d317a018b4" containerID="1f02b0ce6189cd8907a41d130021a435b3401763c018d51364fa691c416f6275" exitCode=0
Nov 03 14:56:30.616973 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:30.616627    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc" event={"ID":"8c5a7822-7874-4929-ae68-b1d317a018b4","Type":"ContainerDied","Data":"1f02b0ce6189cd8907a41d130021a435b3401763c018d51364fa691c416f6275"}
Nov 03 14:56:31.759225 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.759201    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:31.808414 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.808383    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:31.851993 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.851965    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca\") pod \"8c5a7822-7874-4929-ae68-b1d317a018b4\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") "
Nov 03 14:56:31.852122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.852030    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl\" (UniqueName: \"kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl\") pod \"8c5a7822-7874-4929-ae68-b1d317a018b4\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") "
Nov 03 14:56:31.852122 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.852092    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca\") pod \"8c5a7822-7874-4929-ae68-b1d317a018b4\" (UID: \"8c5a7822-7874-4929-ae68-b1d317a018b4\") "
Nov 03 14:56:31.852336 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.852316    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca" pod "8c5a7822-7874-4929-ae68-b1d317a018b4" (UID: "8c5a7822-7874-4929-ae68-b1d317a018b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:56:31.852484 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.852462    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:56:31.854143 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.854112    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl" pod "8c5a7822-7874-4929-ae68-b1d317a018b4" (UID: "8c5a7822-7874-4929-ae68-b1d317a018b4"). InnerVolumeSpecName "kube-api-access-rwgjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:56:31.854212 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.854144    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca" pod "8c5a7822-7874-4929-ae68-b1d317a018b4" (UID: "8c5a7822-7874-4929-ae68-b1d317a018b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:56:31.953703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.953679    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwgjl\" (UniqueName: \"kubernetes.io/projected/8c5a7822-7874-4929-ae68-b1d317a018b4-kube-api-access-rwgjl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:56:31.953703 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:31.953704    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c5a7822-7874-4929-ae68-b1d317a018b4-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:56:32.345601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:32.345521    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:32.628802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:32.628720    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"
Nov 03 14:56:32.628802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:32.628732    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc" event={"ID":"8c5a7822-7874-4929-ae68-b1d317a018b4","Type":"ContainerDied","Data":"f818eeeeea72c98967ec0457169be33951c2dde6319971d5a378063a93ba4639"}
Nov 03 14:56:32.628802 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:32.628801    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f818eeeeea72c98967ec0457169be33951c2dde6319971d5a378063a93ba4639"
Nov 03 14:56:32.877165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:32.877135    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:56:34.399637 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:34.399600    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:36.907028 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:36.906993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:38.930958 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:38.930914    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:39.618066 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:39.618031    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:41.382783 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:41.382752    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:43.424318 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:43.424284    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:45.873770 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:45.873737    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:47.108919 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:47.108883    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:47.659271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:47.659208    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:47.712865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:47.712832    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"]
Nov 03 14:56:47.713058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:47.713029    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj" podUID="c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" containerName="payload-processing" containerID="cri-o://68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363" gracePeriod=30
Nov 03 14:56:47.879302 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:47.879267    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:47.958745 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:47.958716    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:56:48.013547 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.013515    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj\" (UniqueName: \"kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj\") pod \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") "
Nov 03 14:56:48.013739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.013585    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume\") pod \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\" (UID: \"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4\") "
Nov 03 14:56:48.013997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.013967    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume" pod "c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" (UID: "c31e94e1-7ccd-49e0-a98b-2d96c6f77de4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:56:48.015479 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.015456    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj" pod "c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" (UID: "c31e94e1-7ccd-49e0-a98b-2d96c6f77de4"). InnerVolumeSpecName "kube-api-access-chmcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:56:48.084533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.084501    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:48.115191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.115155    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:56:48.115191 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.115187    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmcj\" (UniqueName: \"kubernetes.io/projected/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4-kube-api-access-chmcj\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:56:48.528906 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.528855    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:48.712631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.712603    6481 generic.go:356] "Generic (PLEG): container finished" podID="c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" containerID="68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363" exitCode=0
Nov 03 14:56:48.712808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.712656    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"
Nov 03 14:56:48.712808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.712689    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj" event={"ID":"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4","Type":"ContainerDied","Data":"68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363"}
Nov 03 14:56:48.712808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.712745    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj" event={"ID":"c31e94e1-7ccd-49e0-a98b-2d96c6f77de4","Type":"ContainerDied","Data":"f242a07d5c5c99de87d8740551893ca8b33eba57141d5e410d6e17e3747719a3"}
Nov 03 14:56:48.712808 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.712772    6481 scope.go:122] "RemoveContainer" containerID="68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363"
Nov 03 14:56:48.721592 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.721574    6481 scope.go:122] "RemoveContainer" containerID="68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363"
Nov 03 14:56:48.721844 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:56:48.721828    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363\": container with ID starting with 68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363 not found: ID does not exist" containerID="68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363"
Nov 03 14:56:48.721907 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.721851    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363"} err="failed to get container status \"68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363\": rpc error: code = NotFound desc = could not find container \"68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363\": container with ID starting with 68063a335a5a3ad130c87b5e7991261295aa6ba271336addd573d486c45b7363 not found: ID does not exist"
Nov 03 14:56:48.776250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.776211    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"]
Nov 03 14:56:48.787673 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:48.787621    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-72b663-8665fcd66b-z2bsj"]
Nov 03 14:56:50.086195 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:50.086142    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" path="/var/lib/kubelet/pods/c31e94e1-7ccd-49e0-a98b-2d96c6f77de4/volumes"
Nov 03 14:56:50.086706 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:50.086584    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:56:51.102621 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:51.102591    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:53.141882 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:53.141847    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:54.792965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:54.792932    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:54.999934 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:54.999903    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:55.472954 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:55.472923    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:55.683968 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:55.683938    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:55.803314 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:55.803205    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"]
Nov 03 14:56:55.803747 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:55.803552    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy" containerID="cri-o://bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709" gracePeriod=30
Nov 03 14:56:55.934480 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:55.934451    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:57.716352 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:56:57.716319    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:56:57.986333 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:56:57.986248    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:57:00.161283 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:00.161248    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:01.061918 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.061891    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:57:01.143862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.143792    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.143862 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.143856    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.143889    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.143936    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144058 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.143965    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144064    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:01.144223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144083    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144140    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144223 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144184    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144277    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket\") pod \"5c748297-e5da-49fa-ac16-998070f730bc\" (UID: \"5c748297-e5da-49fa-ac16-998070f730bc\") "
Nov 03 14:57:01.144489 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144424    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:01.144863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144624    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:01.144863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144784    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.144863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144805    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.144863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144820    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.144863 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.144810    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:01.145265 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.145026    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:57:01.146357 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.146325    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:57:01.146481 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.146389    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "kube-api-access-jvqbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:57:01.146561 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.146546    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:57:01.146623 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.146560    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy" pod "5c748297-e5da-49fa-ac16-998070f730bc" (UID: "5c748297-e5da-49fa-ac16-998070f730bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:01.246080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.246054    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c748297-e5da-49fa-ac16-998070f730bc-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.246080 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.246077    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.246485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.246089    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c748297-e5da-49fa-ac16-998070f730bc-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.246485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.246098    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.246485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.246105    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c748297-e5da-49fa-ac16-998070f730bc-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.246485 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.246115    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvqbr\" (UniqueName: \"kubernetes.io/projected/5c748297-e5da-49fa-ac16-998070f730bc-kube-api-access-jvqbr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:01.696603 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.696569    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:01.781019 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.780987    6481 generic.go:356] "Generic (PLEG): container finished" podID="5c748297-e5da-49fa-ac16-998070f730bc" containerID="bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709" exitCode=0
Nov 03 14:57:01.781201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.781037    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"
Nov 03 14:57:01.781201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.781039    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" event={"ID":"5c748297-e5da-49fa-ac16-998070f730bc","Type":"ContainerDied","Data":"bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709"}
Nov 03 14:57:01.781201 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.781169    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c" event={"ID":"5c748297-e5da-49fa-ac16-998070f730bc","Type":"ContainerDied","Data":"f6a96ceb57f785ef49da57abe7bb63f3f8e178f5c5acbb7b115fcbc707fc1ee0"}
Nov 03 14:57:01.781434 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.781210    6481 scope.go:122] "RemoveContainer" containerID="bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709"
Nov 03 14:57:01.791899 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.791878    6481 scope.go:122] "RemoveContainer" containerID="84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"
Nov 03 14:57:01.800963 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.800944    6481 scope.go:122] "RemoveContainer" containerID="bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709"
Nov 03 14:57:01.801257 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:57:01.801224    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709\": container with ID starting with bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709 not found: ID does not exist" containerID="bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709"
Nov 03 14:57:01.801305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.801268    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709"} err="failed to get container status \"bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709\": rpc error: code = NotFound desc = could not find container \"bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709\": container with ID starting with bd6ce9daad7e25210e3a96f3bacd27b203b5f25871d860a87ff567a7440c3709 not found: ID does not exist"
Nov 03 14:57:01.801305 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.801287    6481 scope.go:122] "RemoveContainer" containerID="84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"
Nov 03 14:57:01.801539 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:57:01.801514    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947\": container with ID starting with 84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947 not found: ID does not exist" containerID="84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"
Nov 03 14:57:01.801580 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.801546    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947"} err="failed to get container status \"84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947\": rpc error: code = NotFound desc = could not find container \"84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947\": container with ID starting with 84fb4fe7da0555ed0dd375b3890bc16f4e92067c0cc857ba7d96bc813bcdb947 not found: ID does not exist"
Nov 03 14:57:01.844689 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.844663    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"]
Nov 03 14:57:01.865282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.865258    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-72b663-openshift-default-86c96cd447-4db8c"]
Nov 03 14:57:01.901781 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:01.901758    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:02.066945 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:02.066757    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c748297-e5da-49fa-ac16-998070f730bc" path="/var/lib/kubelet/pods/5c748297-e5da-49fa-ac16-998070f730bc/volumes"
Nov 03 14:57:02.119452 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:02.119425    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:04.587120 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:04.587088    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:06.652670 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:06.652632    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:09.084645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:09.084604    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:09.600966 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:09.600913    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:10.809044 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:10.809012    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:10.870656 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:10.870624    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"]
Nov 03 14:57:10.870854 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:10.870825    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl" podUID="2741c5f4-d9b5-442a-bcee-44cf3a0899c4" containerName="payload-pre-processing" containerID="cri-o://cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08" gracePeriod=30
Nov 03 14:57:11.026885 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.026854    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:11.128562 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.128539    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:57:11.236639 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.236605    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:11.244739 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.244699    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t\" (UniqueName: \"kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t\") pod \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") "
Nov 03 14:57:11.244933 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.244802    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume\") pod \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\" (UID: \"2741c5f4-d9b5-442a-bcee-44cf3a0899c4\") "
Nov 03 14:57:11.245192 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.245158    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume" pod "2741c5f4-d9b5-442a-bcee-44cf3a0899c4" (UID: "2741c5f4-d9b5-442a-bcee-44cf3a0899c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:57:11.246978 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.246949    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t" pod "2741c5f4-d9b5-442a-bcee-44cf3a0899c4" (UID: "2741c5f4-d9b5-442a-bcee-44cf3a0899c4"). InnerVolumeSpecName "kube-api-access-p6j8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:57:11.346675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.346619    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6j8t\" (UniqueName: \"kubernetes.io/projected/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-kube-api-access-p6j8t\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:11.346675 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.346665    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2741c5f4-d9b5-442a-bcee-44cf3a0899c4-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:11.835880 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.835843    6481 generic.go:356] "Generic (PLEG): container finished" podID="2741c5f4-d9b5-442a-bcee-44cf3a0899c4" containerID="cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08" exitCode=0
Nov 03 14:57:11.836333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.835908    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"
Nov 03 14:57:11.836333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.835930    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl" event={"ID":"2741c5f4-d9b5-442a-bcee-44cf3a0899c4","Type":"ContainerDied","Data":"cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08"}
Nov 03 14:57:11.836333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.836019    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl" event={"ID":"2741c5f4-d9b5-442a-bcee-44cf3a0899c4","Type":"ContainerDied","Data":"4b7cec3af8593efb300febe705101801d1473c28bd61abdbb59198466d537c45"}
Nov 03 14:57:11.836333 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.836049    6481 scope.go:122] "RemoveContainer" containerID="cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08"
Nov 03 14:57:11.845248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.845204    6481 scope.go:122] "RemoveContainer" containerID="cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08"
Nov 03 14:57:11.845506 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:57:11.845486    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08\": container with ID starting with cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08 not found: ID does not exist" containerID="cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08"
Nov 03 14:57:11.845548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.845518    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08"} err="failed to get container status \"cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08\": rpc error: code = NotFound desc = could not find container \"cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08\": container with ID starting with cb50937e4b8eb744d14214114952271facb74dc1855b32cff27f482b4a9dcf08 not found: ID does not exist"
Nov 03 14:57:11.899780 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.899760    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"]
Nov 03 14:57:11.910438 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:11.910410    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-131649-58674d76d7-vjhhl"]
Nov 03 14:57:12.067472 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:12.067445    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2741c5f4-d9b5-442a-bcee-44cf3a0899c4" path="/var/lib/kubelet/pods/2741c5f4-d9b5-442a-bcee-44cf3a0899c4/volumes"
Nov 03 14:57:13.103112 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:13.103029    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:57:16.666576 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:16.666545    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:16.886798 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:16.886771    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:17.471930 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:17.471896    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:17.769037 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:17.768954    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"]
Nov 03 14:57:17.769486 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:17.769211    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" containerID="cri-o://db61207fe86b9285b41475729495c09011da8f849f81c53412057dc877c49e12" gracePeriod=30
Nov 03 14:57:17.904096 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:17.904062    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:21.532831 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:21.532798    6481 patch_prober.go:29] interesting pod/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 14:57:21.533214 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:21.532853    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 14:57:22.896833 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:22.896798    6481 generic.go:356] "Generic (PLEG): container finished" podID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerID="db61207fe86b9285b41475729495c09011da8f849f81c53412057dc877c49e12" exitCode=0
Nov 03 14:57:22.897219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:22.896895    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" event={"ID":"c009545a-8a66-4f40-8407-d958bd1df4c6","Type":"ContainerDied","Data":"db61207fe86b9285b41475729495c09011da8f849f81c53412057dc877c49e12"}
Nov 03 14:57:22.897219 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:22.896988    6481 scope.go:122] "RemoveContainer" containerID="ebfd7ab4ba1b109eea487a3eff92f14cab33b7a4f2e6964e97c1f15e8ded0417"
Nov 03 14:57:23.022976 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.022953    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:57:23.167588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167562    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.167772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167622    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.167772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167640    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.167772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167665    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.167772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167694    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.167772 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167715    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.168068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167795    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.168068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167847    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.168068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167880    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:23.168068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.167976    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:23.168068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168020    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert\") pod \"c009545a-8a66-4f40-8407-d958bd1df4c6\" (UID: \"c009545a-8a66-4f40-8407-d958bd1df4c6\") "
Nov 03 14:57:23.168068 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168042    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:23.168403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168305    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:23.168403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168387    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:57:23.168542 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168523    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.168602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168546    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.168602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168559    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c009545a-8a66-4f40-8407-d958bd1df4c6-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.168602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168575    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.168602 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.168589    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.170084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.170062    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 14:57:23.170213 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.170110    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 14:57:23.170524 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.170497    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:57:23.170627 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.170502    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c" pod "c009545a-8a66-4f40-8407-d958bd1df4c6" (UID: "c009545a-8a66-4f40-8407-d958bd1df4c6"). InnerVolumeSpecName "kube-api-access-m5g2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:57:23.270075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.270050    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.270075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.270075    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5g2c\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-kube-api-access-m5g2c\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.270253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.270088    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.270253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.270100    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c009545a-8a66-4f40-8407-d958bd1df4c6-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:57:23.903502 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.903470    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"
Nov 03 14:57:23.903926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.903500    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t" event={"ID":"c009545a-8a66-4f40-8407-d958bd1df4c6","Type":"ContainerDied","Data":"2f109f2eedcc250aa89df9828ea2661722f63be9118741598465b43a7b968b6f"}
Nov 03 14:57:23.903926 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.903585    6481 scope.go:122] "RemoveContainer" containerID="db61207fe86b9285b41475729495c09011da8f849f81c53412057dc877c49e12"
Nov 03 14:57:23.969776 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.969743    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"]
Nov 03 14:57:23.981355 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:23.981328    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-131649-openshift-default-69b48db985-v7j9t"]
Nov 03 14:57:24.065878 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:24.065837    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" path="/var/lib/kubelet/pods/c009545a-8a66-4f40-8407-d958bd1df4c6/volumes"
Nov 03 14:57:24.484184 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:24.484149    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:57:27.985407 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:57:27.985371    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:57:33.704588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.704552    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5985fbdcbc-h9tqm"]
Nov 03 14:57:33.705052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705012    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy"
Nov 03 14:57:33.705052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705023    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy"
Nov 03 14:57:33.705052 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705048    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705055    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705064    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705069    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705078    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8c5a7822-7874-4929-ae68-b1d317a018b4" containerName="revoke-keys"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705083    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c5a7822-7874-4929-ae68-b1d317a018b4" containerName="revoke-keys"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705095    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705099    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705110    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" containerName="payload-processing"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705116    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" containerName="payload-processing"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705128    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2741c5f4-d9b5-442a-bcee-44cf3a0899c4" containerName="payload-pre-processing"
Nov 03 14:57:33.705160 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705132    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2741c5f4-d9b5-442a-bcee-44cf3a0899c4" containerName="payload-pre-processing"
Nov 03 14:57:33.705553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705188    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy"
Nov 03 14:57:33.705553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705198    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy"
Nov 03 14:57:33.705553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705209    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c5a7822-7874-4929-ae68-b1d317a018b4" containerName="revoke-keys"
Nov 03 14:57:33.705553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705216    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c009545a-8a66-4f40-8407-d958bd1df4c6" containerName="istio-proxy"
Nov 03 14:57:33.705553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705222    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c31e94e1-7ccd-49e0-a98b-2d96c6f77de4" containerName="payload-processing"
Nov 03 14:57:33.705553 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.705239    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2741c5f4-d9b5-442a-bcee-44cf3a0899c4" containerName="payload-pre-processing"
Nov 03 14:57:33.708384 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.708368    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:33.729153 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.729130    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5985fbdcbc-h9tqm"]
Nov 03 14:57:33.736041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.736022    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-wrnr9\""
Nov 03 14:57:33.736269 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.736052    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 14:57:33.772282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.772258    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume\") pod \"payload-processing-5985fbdcbc-h9tqm\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") " pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:33.772459 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.772439    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkxl\" (UniqueName: \"kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl\") pod \"payload-processing-5985fbdcbc-h9tqm\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") " pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:33.873528 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.873501    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume\") pod \"payload-processing-5985fbdcbc-h9tqm\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") " pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:33.873634 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.873592    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkxl\" (UniqueName: \"kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl\") pod \"payload-processing-5985fbdcbc-h9tqm\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") " pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:33.874198 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.874178    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume\") pod \"payload-processing-5985fbdcbc-h9tqm\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") " pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:33.895877 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:33.895848    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkxl\" (UniqueName: \"kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl\") pod \"payload-processing-5985fbdcbc-h9tqm\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") " pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:34.021015 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:34.020935    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:34.324156 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:34.324134    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5985fbdcbc-h9tqm"]
Nov 03 14:57:34.326309 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:57:34.326281    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e61446e_543f_417d_af63_5aedaf42858e.slice/crio-e6f3c868a3e9927af9062fc585977778e232311d909591df659cae4244b83028 WatchSource:0}: Error finding container e6f3c868a3e9927af9062fc585977778e232311d909591df659cae4244b83028: Status 404 returned error can't find the container with id e6f3c868a3e9927af9062fc585977778e232311d909591df659cae4244b83028
Nov 03 14:57:34.963072 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:34.963033    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm" event={"ID":"8e61446e-543f-417d-af63-5aedaf42858e","Type":"ContainerStarted","Data":"a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560"}
Nov 03 14:57:34.963493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:34.963093    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm" event={"ID":"8e61446e-543f-417d-af63-5aedaf42858e","Type":"ContainerStarted","Data":"e6f3c868a3e9927af9062fc585977778e232311d909591df659cae4244b83028"}
Nov 03 14:57:34.963493 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:34.963123    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:35.007633 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:35.007571    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm" podStartSLOduration=2.007553341 podStartE2EDuration="2.007553341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:57:35.003973947 +0000 UTC m=+2733.054005130" watchObservedRunningTime="2026-11-03 14:57:35.007553341 +0000 UTC m=+2733.057584525"
Nov 03 14:57:45.970606 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:45.970575    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:57:48.014749 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.014714    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-95b747dcd-rw869"]
Nov 03 14:57:48.015433 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.015413    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c748297-e5da-49fa-ac16-998070f730bc" containerName="istio-proxy"
Nov 03 14:57:48.018613 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.018590    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.039811 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.039785    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-95b747dcd-rw869"]
Nov 03 14:57:48.107061 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.107030    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aff64b7-93ff-44f1-b495-3dab807c820d-plugins-config-volume\") pod \"payload-processing-95b747dcd-rw869\" (UID: \"3aff64b7-93ff-44f1-b495-3dab807c820d\") " pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.107282 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.107092    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xvlh\" (UniqueName: \"kubernetes.io/projected/3aff64b7-93ff-44f1-b495-3dab807c820d-kube-api-access-9xvlh\") pod \"payload-processing-95b747dcd-rw869\" (UID: \"3aff64b7-93ff-44f1-b495-3dab807c820d\") " pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.208039 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.208007    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xvlh\" (UniqueName: \"kubernetes.io/projected/3aff64b7-93ff-44f1-b495-3dab807c820d-kube-api-access-9xvlh\") pod \"payload-processing-95b747dcd-rw869\" (UID: \"3aff64b7-93ff-44f1-b495-3dab807c820d\") " pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.208251 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.208140    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aff64b7-93ff-44f1-b495-3dab807c820d-plugins-config-volume\") pod \"payload-processing-95b747dcd-rw869\" (UID: \"3aff64b7-93ff-44f1-b495-3dab807c820d\") " pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.208871 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.208848    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3aff64b7-93ff-44f1-b495-3dab807c820d-plugins-config-volume\") pod \"payload-processing-95b747dcd-rw869\" (UID: \"3aff64b7-93ff-44f1-b495-3dab807c820d\") " pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.231571 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.231535    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xvlh\" (UniqueName: \"kubernetes.io/projected/3aff64b7-93ff-44f1-b495-3dab807c820d-kube-api-access-9xvlh\") pod \"payload-processing-95b747dcd-rw869\" (UID: \"3aff64b7-93ff-44f1-b495-3dab807c820d\") " pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.331258 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.331156    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:48.631840 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:48.631800    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-95b747dcd-rw869"]
Nov 03 14:57:48.632952 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:57:48.632926    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aff64b7_93ff_44f1_b495_3dab807c820d.slice/crio-842bc0438d32e3e129fbf3559d2e6200e9346c6a7ae211b74cec659a97b894c7 WatchSource:0}: Error finding container 842bc0438d32e3e129fbf3559d2e6200e9346c6a7ae211b74cec659a97b894c7: Status 404 returned error can't find the container with id 842bc0438d32e3e129fbf3559d2e6200e9346c6a7ae211b74cec659a97b894c7
Nov 03 14:57:49.039140 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:49.039104    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-95b747dcd-rw869" event={"ID":"3aff64b7-93ff-44f1-b495-3dab807c820d","Type":"ContainerStarted","Data":"fbcbb4a05ce533da017896cce2343a51885dd39e3414666cc39039f757bd090b"}
Nov 03 14:57:49.039551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:49.039156    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-95b747dcd-rw869" event={"ID":"3aff64b7-93ff-44f1-b495-3dab807c820d","Type":"ContainerStarted","Data":"842bc0438d32e3e129fbf3559d2e6200e9346c6a7ae211b74cec659a97b894c7"}
Nov 03 14:57:49.039551 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:49.039267    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:57:49.079666 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:57:49.079626    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-95b747dcd-rw869" podStartSLOduration=2.079615571 podStartE2EDuration="2.079615571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:57:49.07900022 +0000 UTC m=+2747.129031416" watchObservedRunningTime="2026-11-03 14:57:49.079615571 +0000 UTC m=+2747.129646754"
Nov 03 14:57:57.985056 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:57:57.985020    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:58:00.047650 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:00.047623    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-95b747dcd-rw869"
Nov 03 14:58:00.122625 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:00.122590    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5985fbdcbc-h9tqm"]
Nov 03 14:58:01.899514 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:01.899481    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:58:02.540997 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:02.540956    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:58:02.747296 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:02.747256    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:58:03.516482 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:03.516440    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:58:04.070952 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:04.070923    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:58:05.123249 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.123172    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm" podUID="8e61446e-543f-417d-af63-5aedaf42858e" containerName="payload-processing" containerID="cri-o://a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560" gracePeriod=25
Nov 03 14:58:05.457441 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.457102    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:58:05.481312 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.481281    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume\") pod \"8e61446e-543f-417d-af63-5aedaf42858e\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") "
Nov 03 14:58:05.481588 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.481570    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl\" (UniqueName: \"kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl\") pod \"8e61446e-543f-417d-af63-5aedaf42858e\" (UID: \"8e61446e-543f-417d-af63-5aedaf42858e\") "
Nov 03 14:58:05.482071 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.482029    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume" pod "8e61446e-543f-417d-af63-5aedaf42858e" (UID: "8e61446e-543f-417d-af63-5aedaf42858e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:58:05.486610 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.484181    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl" pod "8e61446e-543f-417d-af63-5aedaf42858e" (UID: "8e61446e-543f-417d-af63-5aedaf42858e"). InnerVolumeSpecName "kube-api-access-jnkxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:58:05.583567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.583521    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e61446e-543f-417d-af63-5aedaf42858e-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:58:05.583567 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:05.583567    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnkxl\" (UniqueName: \"kubernetes.io/projected/8e61446e-543f-417d-af63-5aedaf42858e-kube-api-access-jnkxl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:58:06.162048 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.162007    6481 generic.go:356] "Generic (PLEG): container finished" podID="8e61446e-543f-417d-af63-5aedaf42858e" containerID="a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560" exitCode=0
Nov 03 14:58:06.162961 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.162818    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm"
Nov 03 14:58:06.166619 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.162842    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm" event={"ID":"8e61446e-543f-417d-af63-5aedaf42858e","Type":"ContainerDied","Data":"a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560"}
Nov 03 14:58:06.166817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.166797    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5985fbdcbc-h9tqm" event={"ID":"8e61446e-543f-417d-af63-5aedaf42858e","Type":"ContainerDied","Data":"e6f3c868a3e9927af9062fc585977778e232311d909591df659cae4244b83028"}
Nov 03 14:58:06.167362 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.166942    6481 scope.go:122] "RemoveContainer" containerID="a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560"
Nov 03 14:58:06.269250 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.262409    6481 scope.go:122] "RemoveContainer" containerID="a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560"
Nov 03 14:58:06.270645 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.270615    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5985fbdcbc-h9tqm"]
Nov 03 14:58:06.311273 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:58:06.290335    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560\": container with ID starting with a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560 not found: ID does not exist" containerID="a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560"
Nov 03 14:58:06.311273 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.290386    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560"} err="failed to get container status \"a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560\": rpc error: code = NotFound desc = could not find container \"a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560\": container with ID starting with a98091a313504add5c24d7bc80e4c767d45d4bf014ce940a70cc04a7c2ac1560 not found: ID does not exist"
Nov 03 14:58:06.334252 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:06.313391    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5985fbdcbc-h9tqm"]
Nov 03 14:58:08.113276 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:08.112368    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e61446e-543f-417d-af63-5aedaf42858e" path="/var/lib/kubelet/pods/8e61446e-543f-417d-af63-5aedaf42858e/volumes"
Nov 03 14:58:08.568260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:08.564838    6481 scope.go:122] "RemoveContainer" containerID="c1fc803e33d1862b6e45a874f5b4d9829031d863f5ef9b158d5a1bdef635a9e1"
Nov 03 14:58:10.524253 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:10.516587    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:58:14.052062 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:14.052028    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"]
Nov 03 14:58:14.066508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:14.066479    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-wphgw"]
Nov 03 14:58:16.071615 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:16.071579    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6192ecf5-b450-4ff5-a9c8-efe4c72255c9" path="/var/lib/kubelet/pods/6192ecf5-b450-4ff5-a9c8-efe4c72255c9/volumes"
Nov 03 14:58:27.985664 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:58:27.985632    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:58:35.040444 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:35.040403    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"]
Nov 03 14:58:35.051904 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:35.051870    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-sbtxt"]
Nov 03 14:58:36.065449 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:36.065417    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5882cd8d-1423-434d-9538-5e35ad117da9" path="/var/lib/kubelet/pods/5882cd8d-1423-434d-9538-5e35ad117da9/volumes"
Nov 03 14:58:56.043533 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:56.043457    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"]
Nov 03 14:58:56.054817 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:56.054786    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cafa65a1-b4ngc"]
Nov 03 14:58:56.066271 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:58:56.066221    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdee3c26-dea8-47ff-a826-38ab91455003" path="/var/lib/kubelet/pods/bdee3c26-dea8-47ff-a826-38ab91455003/volumes"
Nov 03 14:58:57.985546 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:58:57.985513    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:59:02.022383 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:02.022343    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:08.756132 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:08.756099    6481 scope.go:122] "RemoveContainer" containerID="685cfcfafa9de9227e9716cc07690f899ed9cde09db95dca5bf936a625d830d6"
Nov 03 14:59:27.610909 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:27.610872    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:27.819129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:27.819098    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:27.985488 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:59:27.985447    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:59:29.746774 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.746742    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"]
Nov 03 14:59:29.747181 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.747171    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8e61446e-543f-417d-af63-5aedaf42858e" containerName="payload-processing"
Nov 03 14:59:29.747260 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.747184    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e61446e-543f-417d-af63-5aedaf42858e" containerName="payload-processing"
Nov 03 14:59:29.747303 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.747279    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e61446e-543f-417d-af63-5aedaf42858e" containerName="payload-processing"
Nov 03 14:59:29.751537 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.751520    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.769823 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.769798    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"]
Nov 03 14:59:29.779339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.779306    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:59:29.779339 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.779310    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:59:29.779508 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.779437    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:59:29.779548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.779518    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:59:29.797075 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.797044    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.797248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.797163    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.797248 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.797218    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p699h\" (UniqueName: \"kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.898313 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.898278    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.898499 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.898449    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.898573 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.898535    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p699h\" (UniqueName: \"kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.898939 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.898919    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.900741 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.900720    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:29.919041 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:29.919019    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p699h\" (UniqueName: \"kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h\") pod \"maas-api-revoke-sub-e2547622-s4ghv\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:30.064215 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:30.064126    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:30.361084 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:30.361058    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"]
Nov 03 14:59:30.363493 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:59:30.363465    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e21dfd9_823f_43fe_8694_79afe874668d.slice/crio-ca2b68133bb9c3b8a25dd597f5471deabe954c8f5b075ac4fab9bb03bfdef349 WatchSource:0}: Error finding container ca2b68133bb9c3b8a25dd597f5471deabe954c8f5b075ac4fab9bb03bfdef349: Status 404 returned error can't find the container with id ca2b68133bb9c3b8a25dd597f5471deabe954c8f5b075ac4fab9bb03bfdef349
Nov 03 14:59:30.642158 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:30.642128    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv" event={"ID":"9e21dfd9-823f-43fe-8694-79afe874668d","Type":"ContainerStarted","Data":"ca2b68133bb9c3b8a25dd597f5471deabe954c8f5b075ac4fab9bb03bfdef349"}
Nov 03 14:59:31.648616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:31.648523    6481 generic.go:356] "Generic (PLEG): container finished" podID="9e21dfd9-823f-43fe-8694-79afe874668d" containerID="8b5d146db50c4f57e9b7dbb43d2d0e12198dd41bc35798800e0211288d03b150" exitCode=0
Nov 03 14:59:31.648616 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:31.648558    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv" event={"ID":"9e21dfd9-823f-43fe-8694-79afe874668d","Type":"ContainerDied","Data":"8b5d146db50c4f57e9b7dbb43d2d0e12198dd41bc35798800e0211288d03b150"}
Nov 03 14:59:32.787971 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.787942    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:32.827069 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.827044    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca\") pod \"9e21dfd9-823f-43fe-8694-79afe874668d\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") "
Nov 03 14:59:32.827224 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.827122    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h\" (UniqueName: \"kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h\") pod \"9e21dfd9-823f-43fe-8694-79afe874668d\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") "
Nov 03 14:59:32.827326 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.827286    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca\") pod \"9e21dfd9-823f-43fe-8694-79afe874668d\" (UID: \"9e21dfd9-823f-43fe-8694-79afe874668d\") "
Nov 03 14:59:32.827651 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.827624    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca" pod "9e21dfd9-823f-43fe-8694-79afe874668d" (UID: "9e21dfd9-823f-43fe-8694-79afe874668d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:59:32.830394 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.830358    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h" pod "9e21dfd9-823f-43fe-8694-79afe874668d" (UID: "9e21dfd9-823f-43fe-8694-79afe874668d"). InnerVolumeSpecName "kube-api-access-p699h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:59:32.830631 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.830603    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca" pod "9e21dfd9-823f-43fe-8694-79afe874668d" (UID: "9e21dfd9-823f-43fe-8694-79afe874668d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:59:32.928690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.928655    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:59:32.928690 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.928688    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p699h\" (UniqueName: \"kubernetes.io/projected/9e21dfd9-823f-43fe-8694-79afe874668d-kube-api-access-p699h\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:59:32.928897 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:32.928699    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e21dfd9-823f-43fe-8694-79afe874668d-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:59:33.660094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:33.660062    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"
Nov 03 14:59:33.660094 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:33.660086    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv" event={"ID":"9e21dfd9-823f-43fe-8694-79afe874668d","Type":"ContainerDied","Data":"ca2b68133bb9c3b8a25dd597f5471deabe954c8f5b075ac4fab9bb03bfdef349"}
Nov 03 14:59:33.660419 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:33.660129    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2b68133bb9c3b8a25dd597f5471deabe954c8f5b075ac4fab9bb03bfdef349"
Nov 03 14:59:40.728174 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:40.728144    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:40.936504 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:40.936466    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:41.149722 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:41.149641    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:42.010575 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:42.010543    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:59:49.079548 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:49.079512    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 14:59:49.789239 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:49.789193    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 14:59:55.598865 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.598830    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"]
Nov 03 14:59:55.599293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.599267    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9e21dfd9-823f-43fe-8694-79afe874668d" containerName="revoke-keys"
Nov 03 14:59:55.599293 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.599279    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e21dfd9-823f-43fe-8694-79afe874668d" containerName="revoke-keys"
Nov 03 14:59:55.599402 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.599392    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e21dfd9-823f-43fe-8694-79afe874668d" containerName="revoke-keys"
Nov 03 14:59:55.603601 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.603581    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.623860 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.623834    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"]
Nov 03 14:59:55.632525 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.632502    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 14:59:55.632652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.632529    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 14:59:55.632652 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.632530    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 14:59:55.632841 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.632816    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 14:59:55.714410 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.714382    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.714555 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.714440    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.714595 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.714551    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnh9\" (UniqueName: \"kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.815035 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.815007    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.815165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.815067    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.815165 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.815107    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnh9\" (UniqueName: \"kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.815665 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.815645    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.817403 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.817383    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.836366 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.836339    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnh9\" (UniqueName: \"kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9\") pod \"maas-api-revoke-sub-3d7041ca-7wgls\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:55.915285 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:55.915267    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:56.211505 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:56.211473    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"]
Nov 03 14:59:56.213424 ip-10-0-128-190 kubenswrapper[6481]: W1103 14:59:56.213394    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0a289b_2ec6_4003_8632_8433a9b4e235.slice/crio-026eab90e15896765e3ac8e6f863a562eaddf5924344d663f54017ab49e0acde WatchSource:0}: Error finding container 026eab90e15896765e3ac8e6f863a562eaddf5924344d663f54017ab49e0acde: Status 404 returned error can't find the container with id 026eab90e15896765e3ac8e6f863a562eaddf5924344d663f54017ab49e0acde
Nov 03 14:59:56.853994 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:56.853961    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls" event={"ID":"8c0a289b-2ec6-4003-8632-8433a9b4e235","Type":"ContainerStarted","Data":"f3d66da3c0f73738345cbf48d5c2a90dad555c9f3c10c0ed2d8b86a65750d6bc"}
Nov 03 14:59:56.854390 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:56.854016    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls" event={"ID":"8c0a289b-2ec6-4003-8632-8433a9b4e235","Type":"ContainerStarted","Data":"026eab90e15896765e3ac8e6f863a562eaddf5924344d663f54017ab49e0acde"}
Nov 03 14:59:56.895614 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:56.895568    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls" podStartSLOduration=1.895553866 podStartE2EDuration="1.895553866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:59:56.894756818 +0000 UTC m=+2874.944788001" watchObservedRunningTime="2026-11-03 14:59:56.895553866 +0000 UTC m=+2874.945585048"
Nov 03 14:59:57.860543 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:57.860508    6481 generic.go:356] "Generic (PLEG): container finished" podID="8c0a289b-2ec6-4003-8632-8433a9b4e235" containerID="f3d66da3c0f73738345cbf48d5c2a90dad555c9f3c10c0ed2d8b86a65750d6bc" exitCode=0
Nov 03 14:59:57.860965 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:57.860592    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls" event={"ID":"8c0a289b-2ec6-4003-8632-8433a9b4e235","Type":"ContainerDied","Data":"f3d66da3c0f73738345cbf48d5c2a90dad555c9f3c10c0ed2d8b86a65750d6bc"}
Nov 03 14:59:57.985781 ip-10-0-128-190 kubenswrapper[6481]: E1103 14:59:57.985744    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 14:59:58.995425 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:58.995398    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:59.149790 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.149756    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca\") pod \"8c0a289b-2ec6-4003-8632-8433a9b4e235\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") "
Nov 03 14:59:59.149989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.149795    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9\" (UniqueName: \"kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9\") pod \"8c0a289b-2ec6-4003-8632-8433a9b4e235\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") "
Nov 03 14:59:59.149989 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.149823    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca\") pod \"8c0a289b-2ec6-4003-8632-8433a9b4e235\" (UID: \"8c0a289b-2ec6-4003-8632-8433a9b4e235\") "
Nov 03 14:59:59.150129 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.150103    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca" pod "8c0a289b-2ec6-4003-8632-8433a9b4e235" (UID: "8c0a289b-2ec6-4003-8632-8433a9b4e235"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 14:59:59.151887 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.151862    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9" pod "8c0a289b-2ec6-4003-8632-8433a9b4e235" (UID: "8c0a289b-2ec6-4003-8632-8433a9b4e235"). InnerVolumeSpecName "kube-api-access-njnh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 14:59:59.152013 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.151923    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca" pod "8c0a289b-2ec6-4003-8632-8433a9b4e235" (UID: "8c0a289b-2ec6-4003-8632-8433a9b4e235"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 14:59:59.251267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.251181    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:59:59.251267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.251208    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnh9\" (UniqueName: \"kubernetes.io/projected/8c0a289b-2ec6-4003-8632-8433a9b4e235-kube-api-access-njnh9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:59:59.251267 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.251218    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0a289b-2ec6-4003-8632-8433a9b4e235-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 14:59:59.871392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.871280    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"
Nov 03 14:59:59.871392 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.871282    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls" event={"ID":"8c0a289b-2ec6-4003-8632-8433a9b4e235","Type":"ContainerDied","Data":"026eab90e15896765e3ac8e6f863a562eaddf5924344d663f54017ab49e0acde"}
Nov 03 14:59:59.891105 ip-10-0-128-190 kubenswrapper[6481]: I1103 14:59:59.871417    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="026eab90e15896765e3ac8e6f863a562eaddf5924344d663f54017ab49e0acde"
Nov 03 15:00:00.157175 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.157151    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"]
Nov 03 15:00:00.157623 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.157608    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8c0a289b-2ec6-4003-8632-8433a9b4e235" containerName="revoke-keys"
Nov 03 15:00:00.157670 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.157624    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0a289b-2ec6-4003-8632-8433a9b4e235" containerName="revoke-keys"
Nov 03 15:00:00.157718 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.157701    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0a289b-2ec6-4003-8632-8433a9b4e235" containerName="revoke-keys"
Nov 03 15:00:00.161054 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.161031    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.180634 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.180608    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"]
Nov 03 15:00:00.189915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.189891    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:00:00.189915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.189891    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:00:00.190114 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.189899    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:00:00.259808 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.259774    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22t72\" (UniqueName: \"kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72\") pod \"maas-api-key-cleanup-29895300-mljh6\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.259962 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.259819    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895300-mljh6\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.360556 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.360523    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22t72\" (UniqueName: \"kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72\") pod \"maas-api-key-cleanup-29895300-mljh6\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.360715 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.360565    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895300-mljh6\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.361159 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.361141    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895300-mljh6\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.381768 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.381729    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22t72\" (UniqueName: \"kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72\") pod \"maas-api-key-cleanup-29895300-mljh6\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.474047 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.473965    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:00.775037 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.775009    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"]
Nov 03 15:00:00.776208 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:00:00.776185    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce7e290_797d_4a5d_bdd3_5d1c36d1ff80.slice/crio-8d413b955b323b9b48dac5101e4098e8e4edbdd3a435302ccfcc4c596621a771 WatchSource:0}: Error finding container 8d413b955b323b9b48dac5101e4098e8e4edbdd3a435302ccfcc4c596621a771: Status 404 returned error can't find the container with id 8d413b955b323b9b48dac5101e4098e8e4edbdd3a435302ccfcc4c596621a771
Nov 03 15:00:00.877743 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:00.877717    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" event={"ID":"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80","Type":"ContainerStarted","Data":"8d413b955b323b9b48dac5101e4098e8e4edbdd3a435302ccfcc4c596621a771"}
Nov 03 15:00:01.884573 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:01.884495    6481 generic.go:356] "Generic (PLEG): container finished" podID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerID="ff10c7e38a537238b9b7703cc1f7a07f3ec67be2a200a9a5d9ec89e5549bf80c" exitCode=60
Nov 03 15:00:01.884573 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:01.884547    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" event={"ID":"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80","Type":"ContainerDied","Data":"ff10c7e38a537238b9b7703cc1f7a07f3ec67be2a200a9a5d9ec89e5549bf80c"}
Nov 03 15:00:01.884984 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:01.884773    6481 scope.go:122] "RemoveContainer" containerID="ff10c7e38a537238b9b7703cc1f7a07f3ec67be2a200a9a5d9ec89e5549bf80c"
Nov 03 15:00:02.119847 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:02.119821    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:00:02.893449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:02.893410    6481 generic.go:356] "Generic (PLEG): container finished" podID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerID="c2b4b9a0ad5621023c612964be25d11ff2d9285152aa7d4959dffc7258fd0388" exitCode=60
Nov 03 15:00:02.893902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:02.893467    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" event={"ID":"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80","Type":"ContainerDied","Data":"c2b4b9a0ad5621023c612964be25d11ff2d9285152aa7d4959dffc7258fd0388"}
Nov 03 15:00:02.893902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:02.893525    6481 scope.go:122] "RemoveContainer" containerID="ff10c7e38a537238b9b7703cc1f7a07f3ec67be2a200a9a5d9ec89e5549bf80c"
Nov 03 15:00:02.893902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:02.893697    6481 scope.go:122] "RemoveContainer" containerID="c2b4b9a0ad5621023c612964be25d11ff2d9285152aa7d4959dffc7258fd0388"
Nov 03 15:00:02.894050 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:00:02.893962    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895300-mljh6_odh-ai-gateway-infra(1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80"
Nov 03 15:00:03.900055 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:03.900027    6481 scope.go:122] "RemoveContainer" containerID="c2b4b9a0ad5621023c612964be25d11ff2d9285152aa7d4959dffc7258fd0388"
Nov 03 15:00:03.900539 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:00:03.900335    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895300-mljh6_odh-ai-gateway-infra(1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80"
Nov 03 15:00:06.986285 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:06.986252    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:00:08.777781 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:08.777750    6481 scope.go:122] "RemoveContainer" containerID="c9450411f21c77325500a60bc5011b76e20b6a4f6ea1932e7bdcce34d8980117"
Nov 03 15:00:08.786139 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:08.786120    6481 scope.go:122] "RemoveContainer" containerID="00b7b9476b3fcdafe5bbe94bb237f771e5f82f0691f1f4a6d665d098869555a5"
Nov 03 15:00:08.793749 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:08.793735    6481 scope.go:122] "RemoveContainer" containerID="a2ce352826c822496814d2b357a92cac7b900de20b36aaecea0060ece3102361"
Nov 03 15:00:11.792971 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:11.792937    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:00:17.060844 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:17.060756    6481 scope.go:122] "RemoveContainer" containerID="c2b4b9a0ad5621023c612964be25d11ff2d9285152aa7d4959dffc7258fd0388"
Nov 03 15:00:18.012258 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:18.012210    6481 generic.go:356] "Generic (PLEG): container finished" podID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerID="15e2e23090a748233c9039fa722afd7121ed5a652430dece10647b6f5be4abf4" exitCode=60
Nov 03 15:00:18.012426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:18.012284    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" event={"ID":"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80","Type":"ContainerDied","Data":"15e2e23090a748233c9039fa722afd7121ed5a652430dece10647b6f5be4abf4"}
Nov 03 15:00:18.012426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:18.012345    6481 scope.go:122] "RemoveContainer" containerID="c2b4b9a0ad5621023c612964be25d11ff2d9285152aa7d4959dffc7258fd0388"
Nov 03 15:00:18.012681 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:18.012665    6481 scope.go:122] "RemoveContainer" containerID="15e2e23090a748233c9039fa722afd7121ed5a652430dece10647b6f5be4abf4"
Nov 03 15:00:18.012979 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:00:18.012953    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895300-mljh6_odh-ai-gateway-infra(1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80"
Nov 03 15:00:19.064648 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.064614    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"]
Nov 03 15:00:19.199744 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.199715    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:19.284646 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.284617    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca\") pod \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") "
Nov 03 15:00:19.284795 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.284720    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72\" (UniqueName: \"kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72\") pod \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\" (UID: \"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80\") "
Nov 03 15:00:19.285025 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.285006    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca" pod "1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" (UID: "1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:00:19.286813 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.286788    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72" pod "1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" (UID: "1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80"). InnerVolumeSpecName "kube-api-access-22t72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:00:19.385694 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.385621    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:00:19.385694 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:19.385649    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22t72\" (UniqueName: \"kubernetes.io/projected/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80-kube-api-access-22t72\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:00:20.025072 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:20.025039    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"
Nov 03 15:00:20.025314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:20.025082    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6" event={"ID":"1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80","Type":"ContainerDied","Data":"8d413b955b323b9b48dac5101e4098e8e4edbdd3a435302ccfcc4c596621a771"}
Nov 03 15:00:20.025314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:20.025149    6481 scope.go:122] "RemoveContainer" containerID="15e2e23090a748233c9039fa722afd7121ed5a652430dece10647b6f5be4abf4"
Nov 03 15:00:20.080415 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:20.080387    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"]
Nov 03 15:00:20.089024 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:20.088995    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-mljh6"]
Nov 03 15:00:22.065744 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:22.065709    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" path="/var/lib/kubelet/pods/1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80/volumes"
Nov 03 15:00:27.985224 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:00:27.985190    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:00:28.396361 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.396280    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:00:28.431592 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.431559    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"]
Nov 03 15:00:28.431964 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.431951    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:28.432010 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.431965    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:28.432010 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.431989    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:28.432010 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.431994    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:28.432108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.432089    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:28.432108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.432102    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:28.436983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.436964    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.456888 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.456859    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"]
Nov 03 15:00:28.466070 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.466051    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:00:28.466322 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.466305    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:00:28.466386 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.466336    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:00:28.466432 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.466416    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:00:28.571314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.571284    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.571506 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.571332    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.571506 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.571455    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsmxs\" (UniqueName: \"kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.654564 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.654488    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:00:28.672017 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.671989    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsmxs\" (UniqueName: \"kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.672187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.672078    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.672187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.672124    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.672892 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.672865    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.674212 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.674188    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.696840 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.696806    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsmxs\" (UniqueName: \"kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs\") pod \"maas-api-revoke-sub-ec843018-zqfbt\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.750520 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.750490    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:28.862750 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:28.862684    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:00:29.065165 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:29.065131    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"]
Nov 03 15:00:29.067188 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:00:29.067159    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43e5756_22cf_43d4_8dff_2c664efa9075.slice/crio-8fd90f7e4761306827cc9d0d2bd64b995f62dc81b9ab3b4dfad4545f04fc1003 WatchSource:0}: Error finding container 8fd90f7e4761306827cc9d0d2bd64b995f62dc81b9ab3b4dfad4545f04fc1003: Status 404 returned error can't find the container with id 8fd90f7e4761306827cc9d0d2bd64b995f62dc81b9ab3b4dfad4545f04fc1003
Nov 03 15:00:30.077117 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:30.077041    6481 generic.go:356] "Generic (PLEG): container finished" podID="d43e5756-22cf-43d4-8dff-2c664efa9075" containerID="be54b3ac833af126fb0192787774768d7f424011b717336773423938d7be758d" exitCode=0
Nov 03 15:00:30.077558 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:30.077123    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt" event={"ID":"d43e5756-22cf-43d4-8dff-2c664efa9075","Type":"ContainerDied","Data":"be54b3ac833af126fb0192787774768d7f424011b717336773423938d7be758d"}
Nov 03 15:00:30.077558 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:30.077181    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt" event={"ID":"d43e5756-22cf-43d4-8dff-2c664efa9075","Type":"ContainerStarted","Data":"8fd90f7e4761306827cc9d0d2bd64b995f62dc81b9ab3b4dfad4545f04fc1003"}
Nov 03 15:00:31.219559 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.219530    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:31.298625 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.298593    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca\") pod \"d43e5756-22cf-43d4-8dff-2c664efa9075\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") "
Nov 03 15:00:31.298808 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.298695    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca\") pod \"d43e5756-22cf-43d4-8dff-2c664efa9075\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") "
Nov 03 15:00:31.298808 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.298747    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs\" (UniqueName: \"kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs\") pod \"d43e5756-22cf-43d4-8dff-2c664efa9075\" (UID: \"d43e5756-22cf-43d4-8dff-2c664efa9075\") "
Nov 03 15:00:31.299012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.298988    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca" pod "d43e5756-22cf-43d4-8dff-2c664efa9075" (UID: "d43e5756-22cf-43d4-8dff-2c664efa9075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:00:31.300762 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.300737    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca" pod "d43e5756-22cf-43d4-8dff-2c664efa9075" (UID: "d43e5756-22cf-43d4-8dff-2c664efa9075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:00:31.300867 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.300783    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs" pod "d43e5756-22cf-43d4-8dff-2c664efa9075" (UID: "d43e5756-22cf-43d4-8dff-2c664efa9075"). InnerVolumeSpecName "kube-api-access-xsmxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:00:31.400170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.400106    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:00:31.400170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.400130    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsmxs\" (UniqueName: \"kubernetes.io/projected/d43e5756-22cf-43d4-8dff-2c664efa9075-kube-api-access-xsmxs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:00:31.400170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:31.400143    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d43e5756-22cf-43d4-8dff-2c664efa9075-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:00:32.087072 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:32.087046    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"
Nov 03 15:00:32.087206 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:32.087080    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt" event={"ID":"d43e5756-22cf-43d4-8dff-2c664efa9075","Type":"ContainerDied","Data":"8fd90f7e4761306827cc9d0d2bd64b995f62dc81b9ab3b4dfad4545f04fc1003"}
Nov 03 15:00:32.087206 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:32.087123    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd90f7e4761306827cc9d0d2bd64b995f62dc81b9ab3b4dfad4545f04fc1003"
Nov 03 15:00:44.357067 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:44.357031    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:00:48.164268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:48.163318    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:00:48.164268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:48.163387    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:00:48.227254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:48.224553    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:00:48.227254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:48.224613    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:00:48.292389 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:48.292355    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:00:48.292652 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:48.292623    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:00:57.323220 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323175    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"]
Nov 03 15:00:57.323808 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323788    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d43e5756-22cf-43d4-8dff-2c664efa9075" containerName="revoke-keys"
Nov 03 15:00:57.323881 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323812    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43e5756-22cf-43d4-8dff-2c664efa9075" containerName="revoke-keys"
Nov 03 15:00:57.323881 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323832    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:57.323881 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323837    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:57.323989 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323938    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43e5756-22cf-43d4-8dff-2c664efa9075" containerName="revoke-keys"
Nov 03 15:00:57.323989 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.323947    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce7e290-797d-4a5d-bdd3-5d1c36d1ff80" containerName="cleanup"
Nov 03 15:00:57.327534 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.327513    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.354420 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.354390    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:00:57.354582 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.354445    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:00:57.354582 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.354517    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:00:57.354721 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.354605    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:00:57.357915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.357886    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"]
Nov 03 15:00:57.429192 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.429148    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.429192 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.429200    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.429417 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.429300    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2x69\" (UniqueName: \"kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.530812 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.530780    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.530812 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.530815    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.531010 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.530837    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2x69\" (UniqueName: \"kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.531482 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.531463    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.533156 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.533137    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.552360 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.552329    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2x69\" (UniqueName: \"kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69\") pod \"maas-api-revoke-sub-046e1343-8l9fm\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.642449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.642380    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:00:57.959999 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:57.959965    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"]
Nov 03 15:00:57.962386 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:00:57.962354    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19124cf0_85ec_41ad_941b_0b793f546267.slice/crio-91a90e9b8e5365b910060e4fcf291680b4091a44cb401020fef78ca3c4002914 WatchSource:0}: Error finding container 91a90e9b8e5365b910060e4fcf291680b4091a44cb401020fef78ca3c4002914: Status 404 returned error can't find the container with id 91a90e9b8e5365b910060e4fcf291680b4091a44cb401020fef78ca3c4002914
Nov 03 15:00:57.988591 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:00:57.988562    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:00:58.288143 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:58.288107    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm" event={"ID":"19124cf0-85ec-41ad-941b-0b793f546267","Type":"ContainerStarted","Data":"486e9bea99596333bbd0903b7e0a76bbf5715f67f95910480a61bd89b5cb5427"}
Nov 03 15:00:58.288307 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:58.288163    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm" event={"ID":"19124cf0-85ec-41ad-941b-0b793f546267","Type":"ContainerStarted","Data":"91a90e9b8e5365b910060e4fcf291680b4091a44cb401020fef78ca3c4002914"}
Nov 03 15:00:58.345643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:58.345592    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm" podStartSLOduration=1.345579002 podStartE2EDuration="1.345579002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:00:58.332764734 +0000 UTC m=+2936.382795918" watchObservedRunningTime="2026-11-03 15:00:58.345579002 +0000 UTC m=+2936.395610184"
Nov 03 15:00:59.294863 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:59.294827    6481 generic.go:356] "Generic (PLEG): container finished" podID="19124cf0-85ec-41ad-941b-0b793f546267" containerID="486e9bea99596333bbd0903b7e0a76bbf5715f67f95910480a61bd89b5cb5427" exitCode=0
Nov 03 15:00:59.295053 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:00:59.294868    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm" event={"ID":"19124cf0-85ec-41ad-941b-0b793f546267","Type":"ContainerDied","Data":"486e9bea99596333bbd0903b7e0a76bbf5715f67f95910480a61bd89b5cb5427"}
Nov 03 15:01:00.423611 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.423580    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:01:00.459511 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.459485    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69\" (UniqueName: \"kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69\") pod \"19124cf0-85ec-41ad-941b-0b793f546267\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") "
Nov 03 15:01:00.459649 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.459596    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca\") pod \"19124cf0-85ec-41ad-941b-0b793f546267\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") "
Nov 03 15:01:00.459723 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.459653    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca\") pod \"19124cf0-85ec-41ad-941b-0b793f546267\" (UID: \"19124cf0-85ec-41ad-941b-0b793f546267\") "
Nov 03 15:01:00.459958 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.459930    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca" pod "19124cf0-85ec-41ad-941b-0b793f546267" (UID: "19124cf0-85ec-41ad-941b-0b793f546267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:01:00.460079 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.459992    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:00.461610 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.461587    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca" pod "19124cf0-85ec-41ad-941b-0b793f546267" (UID: "19124cf0-85ec-41ad-941b-0b793f546267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:01:00.461698 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.461651    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69" pod "19124cf0-85ec-41ad-941b-0b793f546267" (UID: "19124cf0-85ec-41ad-941b-0b793f546267"). InnerVolumeSpecName "kube-api-access-w2x69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:01:00.561042 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.560973    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2x69\" (UniqueName: \"kubernetes.io/projected/19124cf0-85ec-41ad-941b-0b793f546267-kube-api-access-w2x69\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:00.561042 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:00.561001    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19124cf0-85ec-41ad-941b-0b793f546267-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:01.306132 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:01.306103    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"
Nov 03 15:01:01.306323 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:01.306148    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm" event={"ID":"19124cf0-85ec-41ad-941b-0b793f546267","Type":"ContainerDied","Data":"91a90e9b8e5365b910060e4fcf291680b4091a44cb401020fef78ca3c4002914"}
Nov 03 15:01:01.306323 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:01.306197    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91a90e9b8e5365b910060e4fcf291680b4091a44cb401020fef78ca3c4002914"
Nov 03 15:01:02.236174 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:02.236140    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:01:07.695726 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.695683    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"]
Nov 03 15:01:07.696280 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.696137    6481 cpu_manager.go:414] "containerMap: removing container" podUID="19124cf0-85ec-41ad-941b-0b793f546267" containerName="revoke-keys"
Nov 03 15:01:07.696280 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.696150    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19124cf0-85ec-41ad-941b-0b793f546267" containerName="revoke-keys"
Nov 03 15:01:07.696280 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.696270    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="19124cf0-85ec-41ad-941b-0b793f546267" containerName="revoke-keys"
Nov 03 15:01:07.698621 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.698599    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.714804 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.714781    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:01:07.726619 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.726596    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:01:07.726781 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.726615    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:01:07.726889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.726679    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:01:07.729953 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.729926    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"]
Nov 03 15:01:07.834002 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.833975    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.834146 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.834072    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.834146 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.834105    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfkfw\" (UniqueName: \"kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.935018 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.934990    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.935172 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.935039    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.935172 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.935063    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfkfw\" (UniqueName: \"kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.935818 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.935799    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.937375 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.937356    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:07.958674 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:07.958614    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfkfw\" (UniqueName: \"kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw\") pod \"maas-api-revoke-sub-d3fba056-5bhmc\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:08.010967 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:08.010943    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:08.314476 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:08.314441    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"]
Nov 03 15:01:08.316146 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:01:08.316117    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6da399_a62b_4519_a576_416d2d1ef2af.slice/crio-c9e26510fc83d3aa3cf1b135ee0fa5d810d2dba51a9cff44604b598a47f5468a WatchSource:0}: Error finding container c9e26510fc83d3aa3cf1b135ee0fa5d810d2dba51a9cff44604b598a47f5468a: Status 404 returned error can't find the container with id c9e26510fc83d3aa3cf1b135ee0fa5d810d2dba51a9cff44604b598a47f5468a
Nov 03 15:01:08.343955 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:08.343914    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc" event={"ID":"2d6da399-a62b-4519-a576-416d2d1ef2af","Type":"ContainerStarted","Data":"c9e26510fc83d3aa3cf1b135ee0fa5d810d2dba51a9cff44604b598a47f5468a"}
Nov 03 15:01:08.787563 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:08.787531    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:01:09.019731 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:09.019697    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:01:09.354524 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:09.354418    6481 generic.go:356] "Generic (PLEG): container finished" podID="2d6da399-a62b-4519-a576-416d2d1ef2af" containerID="7e27910a2c7fe991fa33ab72527a231b367550e1fd5e3cb66a4d725d3b64a47c" exitCode=0
Nov 03 15:01:09.354843 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:09.354606    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc" event={"ID":"2d6da399-a62b-4519-a576-416d2d1ef2af","Type":"ContainerDied","Data":"7e27910a2c7fe991fa33ab72527a231b367550e1fd5e3cb66a4d725d3b64a47c"}
Nov 03 15:01:10.998185 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:10.998151    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:11.154311 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.152870    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca\") pod \"2d6da399-a62b-4519-a576-416d2d1ef2af\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") "
Nov 03 15:01:11.154311 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.152968    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca\") pod \"2d6da399-a62b-4519-a576-416d2d1ef2af\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") "
Nov 03 15:01:11.154311 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.153047    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw\" (UniqueName: \"kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw\") pod \"2d6da399-a62b-4519-a576-416d2d1ef2af\" (UID: \"2d6da399-a62b-4519-a576-416d2d1ef2af\") "
Nov 03 15:01:11.193701 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.193666    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca" pod "2d6da399-a62b-4519-a576-416d2d1ef2af" (UID: "2d6da399-a62b-4519-a576-416d2d1ef2af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:01:11.196871 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.196841    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw" pod "2d6da399-a62b-4519-a576-416d2d1ef2af" (UID: "2d6da399-a62b-4519-a576-416d2d1ef2af"). InnerVolumeSpecName "kube-api-access-cfkfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:01:11.235382 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.235355    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca" pod "2d6da399-a62b-4519-a576-416d2d1ef2af" (UID: "2d6da399-a62b-4519-a576-416d2d1ef2af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:01:11.283504 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.283478    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:11.283697 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.283682    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfkfw\" (UniqueName: \"kubernetes.io/projected/2d6da399-a62b-4519-a576-416d2d1ef2af-kube-api-access-cfkfw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:11.283847 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.283831    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d6da399-a62b-4519-a576-416d2d1ef2af-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:11.381539 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.381499    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc" event={"ID":"2d6da399-a62b-4519-a576-416d2d1ef2af","Type":"ContainerDied","Data":"c9e26510fc83d3aa3cf1b135ee0fa5d810d2dba51a9cff44604b598a47f5468a"}
Nov 03 15:01:11.382068 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.382038    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9e26510fc83d3aa3cf1b135ee0fa5d810d2dba51a9cff44604b598a47f5468a"
Nov 03 15:01:11.387282 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:11.381837    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"
Nov 03 15:01:19.090169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:19.090137    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:01:23.884958 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:23.884924    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:01:28.035265 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:01:28.035199    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:01:32.041741 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:32.041701    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"]
Nov 03 15:01:32.052057 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:32.052031    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d733532-gwtfc"]
Nov 03 15:01:32.066039 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:32.066007    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c5a7822-7874-4929-ae68-b1d317a018b4" path="/var/lib/kubelet/pods/8c5a7822-7874-4929-ae68-b1d317a018b4/volumes"
Nov 03 15:01:33.519257 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.519195    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"]
Nov 03 15:01:33.519853 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.519833    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2d6da399-a62b-4519-a576-416d2d1ef2af" containerName="revoke-keys"
Nov 03 15:01:33.519952 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.519856    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6da399-a62b-4519-a576-416d2d1ef2af" containerName="revoke-keys"
Nov 03 15:01:33.520016 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.519953    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6da399-a62b-4519-a576-416d2d1ef2af" containerName="revoke-keys"
Nov 03 15:01:33.523177 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.523157    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.539720 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.539699    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:01:33.550666 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.550648    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:01:33.550834 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.550816    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:01:33.550909 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.550846    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:01:33.552873 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.552851    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"]
Nov 03 15:01:33.636278 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.636250    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjc6x\" (UniqueName: \"kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.636390 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.636351    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.636390 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.636385    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.737148 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.737116    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjc6x\" (UniqueName: \"kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.737341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.737210    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.737341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.737258    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.737873 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.737846    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.739476 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.739457    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.757965 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.757938    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjc6x\" (UniqueName: \"kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x\") pod \"maas-api-revoke-sub-ae594979-4rpkq\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:33.836033 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:33.835968    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:34.135468 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:34.135434    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"]
Nov 03 15:01:34.136567 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:01:34.136541    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465161c8_91a0_4363_9d57_5b67d3946dca.slice/crio-de6c6d6186f5b771b22ae2870532ba932503a65abddf413f7853776a1b3e400e WatchSource:0}: Error finding container de6c6d6186f5b771b22ae2870532ba932503a65abddf413f7853776a1b3e400e: Status 404 returned error can't find the container with id de6c6d6186f5b771b22ae2870532ba932503a65abddf413f7853776a1b3e400e
Nov 03 15:01:34.566909 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:34.566875    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq" event={"ID":"465161c8-91a0-4363-9d57-5b67d3946dca","Type":"ContainerStarted","Data":"65a26c333265e038d3202d835522d04e011232a3fc56f06108aa59cc5f3c5a6b"}
Nov 03 15:01:34.567308 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:34.566927    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq" event={"ID":"465161c8-91a0-4363-9d57-5b67d3946dca","Type":"ContainerStarted","Data":"de6c6d6186f5b771b22ae2870532ba932503a65abddf413f7853776a1b3e400e"}
Nov 03 15:01:34.609126 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:34.609082    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq" podStartSLOduration=1.609068494 podStartE2EDuration="1.609068494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:01:34.608300769 +0000 UTC m=+2972.658331946" watchObservedRunningTime="2026-11-03 15:01:34.609068494 +0000 UTC m=+2972.659099668"
Nov 03 15:01:35.573884 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:35.573843    6481 generic.go:356] "Generic (PLEG): container finished" podID="465161c8-91a0-4363-9d57-5b67d3946dca" containerID="65a26c333265e038d3202d835522d04e011232a3fc56f06108aa59cc5f3c5a6b" exitCode=0
Nov 03 15:01:35.574310 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:35.573910    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq" event={"ID":"465161c8-91a0-4363-9d57-5b67d3946dca","Type":"ContainerDied","Data":"65a26c333265e038d3202d835522d04e011232a3fc56f06108aa59cc5f3c5a6b"}
Nov 03 15:01:36.721433 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.721407    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:36.869976 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.869879    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca\") pod \"465161c8-91a0-4363-9d57-5b67d3946dca\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") "
Nov 03 15:01:36.869976 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.869929    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x\" (UniqueName: \"kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x\") pod \"465161c8-91a0-4363-9d57-5b67d3946dca\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") "
Nov 03 15:01:36.869976 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.869973    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca\") pod \"465161c8-91a0-4363-9d57-5b67d3946dca\" (UID: \"465161c8-91a0-4363-9d57-5b67d3946dca\") "
Nov 03 15:01:36.870319 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.870287    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca" pod "465161c8-91a0-4363-9d57-5b67d3946dca" (UID: "465161c8-91a0-4363-9d57-5b67d3946dca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:01:36.872124 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.872103    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x" pod "465161c8-91a0-4363-9d57-5b67d3946dca" (UID: "465161c8-91a0-4363-9d57-5b67d3946dca"). InnerVolumeSpecName "kube-api-access-mjc6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:01:36.872193 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.872122    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca" pod "465161c8-91a0-4363-9d57-5b67d3946dca" (UID: "465161c8-91a0-4363-9d57-5b67d3946dca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:01:36.971339 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.971303    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:36.971339 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.971333    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjc6x\" (UniqueName: \"kubernetes.io/projected/465161c8-91a0-4363-9d57-5b67d3946dca-kube-api-access-mjc6x\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:36.971339 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:36.971343    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/465161c8-91a0-4363-9d57-5b67d3946dca-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:37.585065 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:37.585041    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"
Nov 03 15:01:37.585065 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:37.585059    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq" event={"ID":"465161c8-91a0-4363-9d57-5b67d3946dca","Type":"ContainerDied","Data":"de6c6d6186f5b771b22ae2870532ba932503a65abddf413f7853776a1b3e400e"}
Nov 03 15:01:37.585319 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:37.585101    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6c6d6186f5b771b22ae2870532ba932503a65abddf413f7853776a1b3e400e"
Nov 03 15:01:44.891788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:44.891704    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:01:49.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.056602    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"]
Nov 03 15:01:49.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.057617    6481 cpu_manager.go:414] "containerMap: removing container" podUID="465161c8-91a0-4363-9d57-5b67d3946dca" containerName="revoke-keys"
Nov 03 15:01:49.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.057639    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="465161c8-91a0-4363-9d57-5b67d3946dca" containerName="revoke-keys"
Nov 03 15:01:49.063252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.057961    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="465161c8-91a0-4363-9d57-5b67d3946dca" containerName="revoke-keys"
Nov 03 15:01:49.077627 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.072479    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.080733 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.080706    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"]
Nov 03 15:01:49.109214 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.108526    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:01:49.109214 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.108612    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:01:49.109214 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.108681    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:01:49.109214 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.108688    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:01:49.194776 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.194732    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5ppn\" (UniqueName: \"kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.195112 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.195079    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.195272 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.195134    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.299334 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.299291    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.300472 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.300447    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.301210 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.301188    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5ppn\" (UniqueName: \"kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.301706 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.300356    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.305248 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.303868    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.324462 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.324398    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5ppn\" (UniqueName: \"kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn\") pod \"maas-api-revoke-sub-a4aef7d7-ff8kl\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.408426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.408385    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:49.976336 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:49.972071    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"]
Nov 03 15:01:50.048629 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:01:50.046424    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1894d814_757b_4aff_944d_0faea157b49c.slice/crio-f298c065502cb28c3f9fb3381bdc674ed7b0c113a5e5c923f6678a36c4581aa1 WatchSource:0}: Error finding container f298c065502cb28c3f9fb3381bdc674ed7b0c113a5e5c923f6678a36c4581aa1: Status 404 returned error can't find the container with id f298c065502cb28c3f9fb3381bdc674ed7b0c113a5e5c923f6678a36c4581aa1
Nov 03 15:01:50.473163 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:50.473131    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:01:50.713036 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:50.712123    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl" event={"ID":"1894d814-757b-4aff-944d-0faea157b49c","Type":"ContainerStarted","Data":"ef6b8ee95df0c6d885431ece1673b6dbf2dae1fc1f623b805fee6ee80b73eadb"}
Nov 03 15:01:50.713036 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:50.712189    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl" event={"ID":"1894d814-757b-4aff-944d-0faea157b49c","Type":"ContainerStarted","Data":"f298c065502cb28c3f9fb3381bdc674ed7b0c113a5e5c923f6678a36c4581aa1"}
Nov 03 15:01:50.760206 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:50.760151    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl" podStartSLOduration=1.760131697 podStartE2EDuration="1.760131697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:01:50.75855229 +0000 UTC m=+2988.808583475" watchObservedRunningTime="2026-11-03 15:01:50.760131697 +0000 UTC m=+2988.810162882"
Nov 03 15:01:51.738727 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:51.738691    6481 generic.go:356] "Generic (PLEG): container finished" podID="1894d814-757b-4aff-944d-0faea157b49c" containerID="ef6b8ee95df0c6d885431ece1673b6dbf2dae1fc1f623b805fee6ee80b73eadb" exitCode=0
Nov 03 15:01:51.739423 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:51.739397    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl" event={"ID":"1894d814-757b-4aff-944d-0faea157b49c","Type":"ContainerDied","Data":"ef6b8ee95df0c6d885431ece1673b6dbf2dae1fc1f623b805fee6ee80b73eadb"}
Nov 03 15:01:52.873521 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.873496    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:52.950299 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.950270    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca\") pod \"1894d814-757b-4aff-944d-0faea157b49c\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") "
Nov 03 15:01:52.950299 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.950308    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn\" (UniqueName: \"kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn\") pod \"1894d814-757b-4aff-944d-0faea157b49c\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") "
Nov 03 15:01:52.950569 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.950411    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca\") pod \"1894d814-757b-4aff-944d-0faea157b49c\" (UID: \"1894d814-757b-4aff-944d-0faea157b49c\") "
Nov 03 15:01:52.950834 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.950809    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca" pod "1894d814-757b-4aff-944d-0faea157b49c" (UID: "1894d814-757b-4aff-944d-0faea157b49c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:01:52.952412 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.952389    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca" pod "1894d814-757b-4aff-944d-0faea157b49c" (UID: "1894d814-757b-4aff-944d-0faea157b49c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:01:52.952565 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:52.952547    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn" pod "1894d814-757b-4aff-944d-0faea157b49c" (UID: "1894d814-757b-4aff-944d-0faea157b49c"). InnerVolumeSpecName "kube-api-access-b5ppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:01:53.051471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:53.051406    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1894d814-757b-4aff-944d-0faea157b49c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:53.051471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:53.051431    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1894d814-757b-4aff-944d-0faea157b49c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:53.051471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:53.051441    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5ppn\" (UniqueName: \"kubernetes.io/projected/1894d814-757b-4aff-944d-0faea157b49c-kube-api-access-b5ppn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:01:53.750861 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:53.750823    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl" event={"ID":"1894d814-757b-4aff-944d-0faea157b49c","Type":"ContainerDied","Data":"f298c065502cb28c3f9fb3381bdc674ed7b0c113a5e5c923f6678a36c4581aa1"}
Nov 03 15:01:53.750861 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:53.750844    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"
Nov 03 15:01:53.751075 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:53.750908    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f298c065502cb28c3f9fb3381bdc674ed7b0c113a5e5c923f6678a36c4581aa1"
Nov 03 15:01:55.175832 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:01:55.175798    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:01:57.985340 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:01:57.985303    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:02:02.415514 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:02.415474    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:02:04.881511 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:04.881476    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:02:06.090489 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.090454    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"]
Nov 03 15:02:06.091676 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.091646    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1894d814-757b-4aff-944d-0faea157b49c" containerName="revoke-keys"
Nov 03 15:02:06.091676 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.091677    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1894d814-757b-4aff-944d-0faea157b49c" containerName="revoke-keys"
Nov 03 15:02:06.091905 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.091835    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1894d814-757b-4aff-944d-0faea157b49c" containerName="revoke-keys"
Nov 03 15:02:06.095518 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.095491    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.126520 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.126487    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:02:06.126694 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.126623    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:02:06.126694 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.126691    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:02:06.126901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.126776    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:02:06.128060 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.127292    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"]
Nov 03 15:02:06.182746 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.182620    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.182746 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.182667    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pft\" (UniqueName: \"kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.182998 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.182858    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.283601 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.283556    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.283810 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.283705    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.283810 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.283734    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pft\" (UniqueName: \"kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.284654 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.284622    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.286702 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.286675    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.306220 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.306182    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pft\" (UniqueName: \"kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft\") pod \"maas-api-revoke-sub-79805ca9-7q7w4\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.433604 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.433564    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:06.798306 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.797948    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"]
Nov 03 15:02:06.872252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:06.862266    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4" event={"ID":"e952a133-9bdb-4af5-8b16-38df266c53cd","Type":"ContainerStarted","Data":"e710e620ebbb14268335e3ec412fc143f97509b4ff0e2c9ebd865f5ead2ae01e"}
Nov 03 15:02:07.890259 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:07.889284    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4" event={"ID":"e952a133-9bdb-4af5-8b16-38df266c53cd","Type":"ContainerStarted","Data":"3229f685ca7ebda8f10ac4dcddf26b853251a690c1318c7d3b02a17506f5f21b"}
Nov 03 15:02:08.914893 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:08.914851    6481 generic.go:356] "Generic (PLEG): container finished" podID="e952a133-9bdb-4af5-8b16-38df266c53cd" containerID="3229f685ca7ebda8f10ac4dcddf26b853251a690c1318c7d3b02a17506f5f21b" exitCode=0
Nov 03 15:02:08.915572 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:08.915547    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4" event={"ID":"e952a133-9bdb-4af5-8b16-38df266c53cd","Type":"ContainerDied","Data":"3229f685ca7ebda8f10ac4dcddf26b853251a690c1318c7d3b02a17506f5f21b"}
Nov 03 15:02:10.057653 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.057630    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:10.144534 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.144504    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca\") pod \"e952a133-9bdb-4af5-8b16-38df266c53cd\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") "
Nov 03 15:02:10.144701 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.144565    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft\" (UniqueName: \"kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft\") pod \"e952a133-9bdb-4af5-8b16-38df266c53cd\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") "
Nov 03 15:02:10.144701 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.144607    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca\") pod \"e952a133-9bdb-4af5-8b16-38df266c53cd\" (UID: \"e952a133-9bdb-4af5-8b16-38df266c53cd\") "
Nov 03 15:02:10.145001 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.144972    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca" pod "e952a133-9bdb-4af5-8b16-38df266c53cd" (UID: "e952a133-9bdb-4af5-8b16-38df266c53cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:02:10.146649 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.146628    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft" pod "e952a133-9bdb-4af5-8b16-38df266c53cd" (UID: "e952a133-9bdb-4af5-8b16-38df266c53cd"). InnerVolumeSpecName "kube-api-access-n2pft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:02:10.146649 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.146638    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca" pod "e952a133-9bdb-4af5-8b16-38df266c53cd" (UID: "e952a133-9bdb-4af5-8b16-38df266c53cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:02:10.245492 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.245432    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:10.245492 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.245455    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2pft\" (UniqueName: \"kubernetes.io/projected/e952a133-9bdb-4af5-8b16-38df266c53cd-kube-api-access-n2pft\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:10.245492 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.245465    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e952a133-9bdb-4af5-8b16-38df266c53cd-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:10.698048 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.698015    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:02:10.933983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.933938    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4" event={"ID":"e952a133-9bdb-4af5-8b16-38df266c53cd","Type":"ContainerDied","Data":"e710e620ebbb14268335e3ec412fc143f97509b4ff0e2c9ebd865f5ead2ae01e"}
Nov 03 15:02:10.934307 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.934285    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e710e620ebbb14268335e3ec412fc143f97509b4ff0e2c9ebd865f5ead2ae01e"
Nov 03 15:02:10.934507 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:10.934494    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"
Nov 03 15:02:18.492747 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:18.492714    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:02:19.793970 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.793921    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"]
Nov 03 15:02:19.795178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.795144    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e952a133-9bdb-4af5-8b16-38df266c53cd" containerName="revoke-keys"
Nov 03 15:02:19.795178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.795179    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e952a133-9bdb-4af5-8b16-38df266c53cd" containerName="revoke-keys"
Nov 03 15:02:19.795415 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.795362    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e952a133-9bdb-4af5-8b16-38df266c53cd" containerName="revoke-keys"
Nov 03 15:02:19.798782 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.798759    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.830455 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.830427    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:02:19.830780 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.830761    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:02:19.830943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.830929    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:02:19.831148 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.831132    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:02:19.837249 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.837203    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"]
Nov 03 15:02:19.838447 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.838420    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gn4l\" (UniqueName: \"kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.838580 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.838475    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.838655 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.838635    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.940001 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.939965    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gn4l\" (UniqueName: \"kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.940223 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.940026    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.940223 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.940110    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.941411 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.941383    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.943171 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.943140    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:19.961892 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:19.961856    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gn4l\" (UniqueName: \"kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l\") pod \"maas-api-revoke-sub-0b7e8edb-tfvlc\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:20.118263 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:20.118137    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:20.487438 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:20.483105    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"]
Nov 03 15:02:21.033253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:21.017371    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc" event={"ID":"1d92703d-4625-4afc-8794-810aa5a5abbe","Type":"ContainerStarted","Data":"1d40f2cea51dadce26d695cec5ec73d25cbef38dbb2d495fd5a3aaaa6f521c3e"}
Nov 03 15:02:22.043314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:22.039984    6481 generic.go:356] "Generic (PLEG): container finished" podID="1d92703d-4625-4afc-8794-810aa5a5abbe" containerID="106aa83dea016071ee5e1add5d115801612156152bf30487478d2ba0761d7df9" exitCode=0
Nov 03 15:02:22.043314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:22.040039    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc" event={"ID":"1d92703d-4625-4afc-8794-810aa5a5abbe","Type":"ContainerDied","Data":"106aa83dea016071ee5e1add5d115801612156152bf30487478d2ba0761d7df9"}
Nov 03 15:02:23.270696 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.270674    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:23.327327 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.327222    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l\" (UniqueName: \"kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l\") pod \"1d92703d-4625-4afc-8794-810aa5a5abbe\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") "
Nov 03 15:02:23.327497 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.327355    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca\") pod \"1d92703d-4625-4afc-8794-810aa5a5abbe\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") "
Nov 03 15:02:23.327567 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.327493    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca\") pod \"1d92703d-4625-4afc-8794-810aa5a5abbe\" (UID: \"1d92703d-4625-4afc-8794-810aa5a5abbe\") "
Nov 03 15:02:23.327899 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.327750    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca" pod "1d92703d-4625-4afc-8794-810aa5a5abbe" (UID: "1d92703d-4625-4afc-8794-810aa5a5abbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:02:23.327899 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.327862    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:23.329659 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.329631    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca" pod "1d92703d-4625-4afc-8794-810aa5a5abbe" (UID: "1d92703d-4625-4afc-8794-810aa5a5abbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:02:23.329872 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.329843    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l" pod "1d92703d-4625-4afc-8794-810aa5a5abbe" (UID: "1d92703d-4625-4afc-8794-810aa5a5abbe"). InnerVolumeSpecName "kube-api-access-5gn4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:02:23.429350 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.429318    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d92703d-4625-4afc-8794-810aa5a5abbe-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:23.429350 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:23.429345    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gn4l\" (UniqueName: \"kubernetes.io/projected/1d92703d-4625-4afc-8794-810aa5a5abbe-kube-api-access-5gn4l\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:24.080334 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:24.080305    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc" event={"ID":"1d92703d-4625-4afc-8794-810aa5a5abbe","Type":"ContainerDied","Data":"1d40f2cea51dadce26d695cec5ec73d25cbef38dbb2d495fd5a3aaaa6f521c3e"}
Nov 03 15:02:24.080471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:24.080344    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"
Nov 03 15:02:24.080471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:24.080351    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d40f2cea51dadce26d695cec5ec73d25cbef38dbb2d495fd5a3aaaa6f521c3e"
Nov 03 15:02:24.376686 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:24.376612    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:02:27.985881 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:02:27.985836    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:02:30.646952 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.646922    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"]
Nov 03 15:02:30.647382 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.647367    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1d92703d-4625-4afc-8794-810aa5a5abbe" containerName="revoke-keys"
Nov 03 15:02:30.647382 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.647379    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d92703d-4625-4afc-8794-810aa5a5abbe" containerName="revoke-keys"
Nov 03 15:02:30.647459 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.647445    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d92703d-4625-4afc-8794-810aa5a5abbe" containerName="revoke-keys"
Nov 03 15:02:30.650956 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.650939    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.677863 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.677840    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:02:30.678040 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.677948    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:02:30.678108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.678040    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:02:30.678165 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.678129    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:02:30.680762 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.680736    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"]
Nov 03 15:02:30.693826 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.693799    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.693986 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.693845    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.694079 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.694052    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k96vn\" (UniqueName: \"kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.794635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.794605    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k96vn\" (UniqueName: \"kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.794797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.794658    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.794797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.794697    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.795462 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.795440    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.797132 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.797109    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.817080 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.817056    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k96vn\" (UniqueName: \"kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn\") pod \"maas-api-revoke-sub-3d4156f1-8ks89\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:30.965314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:30.965281    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:31.267475 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:31.267450    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"]
Nov 03 15:02:31.268564 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:02:31.268536    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b6577b_59dc_440f_b645_5096655100ef.slice/crio-1bb4d1ed8737780d242423016e0aefc6cf3f0ca5d81ebfd145c09f53dd31a6d8 WatchSource:0}: Error finding container 1bb4d1ed8737780d242423016e0aefc6cf3f0ca5d81ebfd145c09f53dd31a6d8: Status 404 returned error can't find the container with id 1bb4d1ed8737780d242423016e0aefc6cf3f0ca5d81ebfd145c09f53dd31a6d8
Nov 03 15:02:32.123171 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:32.123140    6481 generic.go:356] "Generic (PLEG): container finished" podID="95b6577b-59dc-440f-b645-5096655100ef" containerID="c7a91a4f74b8b93e2530b069a1065517c44beed03b59656ba5d798595d0c7a04" exitCode=0
Nov 03 15:02:32.123549 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:32.123218    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89" event={"ID":"95b6577b-59dc-440f-b645-5096655100ef","Type":"ContainerDied","Data":"c7a91a4f74b8b93e2530b069a1065517c44beed03b59656ba5d798595d0c7a04"}
Nov 03 15:02:32.123549 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:32.123286    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89" event={"ID":"95b6577b-59dc-440f-b645-5096655100ef","Type":"ContainerStarted","Data":"1bb4d1ed8737780d242423016e0aefc6cf3f0ca5d81ebfd145c09f53dd31a6d8"}
Nov 03 15:02:33.264752 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.264730    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:33.318656 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.318617    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca\") pod \"95b6577b-59dc-440f-b645-5096655100ef\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") "
Nov 03 15:02:33.318805 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.318708    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca\") pod \"95b6577b-59dc-440f-b645-5096655100ef\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") "
Nov 03 15:02:33.318805 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.318766    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn\" (UniqueName: \"kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn\") pod \"95b6577b-59dc-440f-b645-5096655100ef\" (UID: \"95b6577b-59dc-440f-b645-5096655100ef\") "
Nov 03 15:02:33.319078 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.319057    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca" pod "95b6577b-59dc-440f-b645-5096655100ef" (UID: "95b6577b-59dc-440f-b645-5096655100ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:02:33.320783 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.320760    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca" pod "95b6577b-59dc-440f-b645-5096655100ef" (UID: "95b6577b-59dc-440f-b645-5096655100ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:02:33.320879 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.320796    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn" pod "95b6577b-59dc-440f-b645-5096655100ef" (UID: "95b6577b-59dc-440f-b645-5096655100ef"). InnerVolumeSpecName "kube-api-access-k96vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:02:33.420012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.419986    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k96vn\" (UniqueName: \"kubernetes.io/projected/95b6577b-59dc-440f-b645-5096655100ef-kube-api-access-k96vn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:33.420012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.420013    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b6577b-59dc-440f-b645-5096655100ef-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:33.420189 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:33.420023    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b6577b-59dc-440f-b645-5096655100ef-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:34.134512 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:34.134481    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"
Nov 03 15:02:34.134650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:34.134508    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89" event={"ID":"95b6577b-59dc-440f-b645-5096655100ef","Type":"ContainerDied","Data":"1bb4d1ed8737780d242423016e0aefc6cf3f0ca5d81ebfd145c09f53dd31a6d8"}
Nov 03 15:02:34.134650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:34.134555    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb4d1ed8737780d242423016e0aefc6cf3f0ca5d81ebfd145c09f53dd31a6d8"
Nov 03 15:02:36.941988 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:36.941954    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:02:42.995765 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:42.995730    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:02:47.283271 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.283192    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"]
Nov 03 15:02:47.293315 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.293286    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8ks89"]
Nov 03 15:02:47.356308 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.356278    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"]
Nov 03 15:02:47.357097 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.357078    6481 cpu_manager.go:414] "containerMap: removing container" podUID="95b6577b-59dc-440f-b645-5096655100ef" containerName="revoke-keys"
Nov 03 15:02:47.357312 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.357272    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b6577b-59dc-440f-b645-5096655100ef" containerName="revoke-keys"
Nov 03 15:02:47.357612 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.357598    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b6577b-59dc-440f-b645-5096655100ef" containerName="revoke-keys"
Nov 03 15:02:47.361597 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.361578    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.380642 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.380454    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"]
Nov 03 15:02:47.381015 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.380979    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:02:47.388502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.386954    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:02:47.388502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.387070    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:02:47.388502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.387141    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:02:47.471286 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.467802    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.471668 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.471638    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.471878 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.471864    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qrkr\" (UniqueName: \"kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.573723 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.573611    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qrkr\" (UniqueName: \"kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.573928 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.573720    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.573928 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.573805    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.574732 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.574702    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.581711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.581679    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.596964 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.596924    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qrkr\" (UniqueName: \"kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr\") pod \"maas-api-revoke-sub-3d4156f1-p2stb\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:47.696603 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:47.696563    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:48.027295 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:48.027205    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"]
Nov 03 15:02:48.028767 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:02:48.028734    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0d1e6e_f1c5_47cb_b618_8a1cc419dc7e.slice/crio-8b68a4f07cc9ee3f8a351e6054f9291b0a49611baba2743daed8806d9d910709 WatchSource:0}: Error finding container 8b68a4f07cc9ee3f8a351e6054f9291b0a49611baba2743daed8806d9d910709: Status 404 returned error can't find the container with id 8b68a4f07cc9ee3f8a351e6054f9291b0a49611baba2743daed8806d9d910709
Nov 03 15:02:48.070176 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:48.070144    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b6577b-59dc-440f-b645-5096655100ef" path="/var/lib/kubelet/pods/95b6577b-59dc-440f-b645-5096655100ef/volumes"
Nov 03 15:02:48.284803 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:48.284704    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb" event={"ID":"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e","Type":"ContainerStarted","Data":"8b68a4f07cc9ee3f8a351e6054f9291b0a49611baba2743daed8806d9d910709"}
Nov 03 15:02:49.311262 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:49.304779    6481 generic.go:356] "Generic (PLEG): container finished" podID="1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" containerID="cb93ff0e14eea3bbf53460d3b1aee3d4c0b75cdc0a2992a30cd48ff28fd277d8" exitCode=0
Nov 03 15:02:49.311262 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:49.304916    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb" event={"ID":"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e","Type":"ContainerDied","Data":"cb93ff0e14eea3bbf53460d3b1aee3d4c0b75cdc0a2992a30cd48ff28fd277d8"}
Nov 03 15:02:50.088267 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:50.087989    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:02:51.339247 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.337377    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:51.388064 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.388026    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb" event={"ID":"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e","Type":"ContainerDied","Data":"8b68a4f07cc9ee3f8a351e6054f9291b0a49611baba2743daed8806d9d910709"}
Nov 03 15:02:51.392942 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.392543    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b68a4f07cc9ee3f8a351e6054f9291b0a49611baba2743daed8806d9d910709"
Nov 03 15:02:51.392942 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.391197    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"
Nov 03 15:02:51.481628 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.481591    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca\") pod \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") "
Nov 03 15:02:51.481941 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.481916    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca\") pod \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") "
Nov 03 15:02:51.482176 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.482157    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr\" (UniqueName: \"kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr\") pod \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\" (UID: \"1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e\") "
Nov 03 15:02:51.483425 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.483402    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca" pod "1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" (UID: "1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:02:51.485445 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.485420    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca" pod "1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" (UID: "1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:02:51.485598 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.485581    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr" pod "1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" (UID: "1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e"). InnerVolumeSpecName "kube-api-access-9qrkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:02:51.583695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.583614    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:51.583695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.583645    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:51.583695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:51.583655    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qrkr\" (UniqueName: \"kubernetes.io/projected/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e-kube-api-access-9qrkr\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:02:57.985426 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:02:57.985393    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:02:59.561749 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:02:59.561709    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:03:02.482345 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:02.482317    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:03:05.109147 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:05.109103    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:03:08.893159 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:08.893123    6481 scope.go:122] "RemoveContainer" containerID="1f02b0ce6189cd8907a41d130021a435b3401763c018d51364fa691c416f6275"
Nov 03 15:03:11.801145 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:11.801110    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:03:18.346789 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.346706    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"]
Nov 03 15:03:18.347162 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.347112    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" containerName="revoke-keys"
Nov 03 15:03:18.347162 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.347124    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" containerName="revoke-keys"
Nov 03 15:03:18.347268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.347249    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" containerName="revoke-keys"
Nov 03 15:03:18.351638 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.351615    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.371412 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.371391    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"]
Nov 03 15:03:18.379326 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.379304    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:03:18.379468 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.379452    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:03:18.379540 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.379515    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:03:18.379676 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.379660    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:03:18.511405 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.511375    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smc7x\" (UniqueName: \"kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.511604 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.511451    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.511674 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.511631    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.613059 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.612978    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.613259 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.613091    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.613259 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.613123    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smc7x\" (UniqueName: \"kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.613711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.613689    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.615217 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.615197    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.633800 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.633773    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smc7x\" (UniqueName: \"kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x\") pod \"maas-api-revoke-sub-e2fc9c5b-vpgbb\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.664423 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.664390    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:18.964526 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:18.964462    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"]
Nov 03 15:03:18.965720 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:03:18.965692    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23e89c6_7340_442f_9e94_945e784275f4.slice/crio-c72acfee6a2596430591739b313c05979ef76398784130d4082a20094263f28d WatchSource:0}: Error finding container c72acfee6a2596430591739b313c05979ef76398784130d4082a20094263f28d: Status 404 returned error can't find the container with id c72acfee6a2596430591739b313c05979ef76398784130d4082a20094263f28d
Nov 03 15:03:19.615790 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:19.615753    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb" event={"ID":"f23e89c6-7340-442f-9e94-945e784275f4","Type":"ContainerStarted","Data":"b8d88ab2bf3cedb2844e3d8464020015f18b71c7f2c20ce340c619feed2663c3"}
Nov 03 15:03:19.616217 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:19.615808    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb" event={"ID":"f23e89c6-7340-442f-9e94-945e784275f4","Type":"ContainerStarted","Data":"c72acfee6a2596430591739b313c05979ef76398784130d4082a20094263f28d"}
Nov 03 15:03:19.656452 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:19.656408    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb" podStartSLOduration=1.656395153 podStartE2EDuration="1.656395153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:03:19.654983767 +0000 UTC m=+3077.705014951" watchObservedRunningTime="2026-11-03 15:03:19.656395153 +0000 UTC m=+3077.706426333"
Nov 03 15:03:20.622076 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:20.622039    6481 generic.go:356] "Generic (PLEG): container finished" podID="f23e89c6-7340-442f-9e94-945e784275f4" containerID="b8d88ab2bf3cedb2844e3d8464020015f18b71c7f2c20ce340c619feed2663c3" exitCode=0
Nov 03 15:03:20.622496 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:20.622121    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb" event={"ID":"f23e89c6-7340-442f-9e94-945e784275f4","Type":"ContainerDied","Data":"b8d88ab2bf3cedb2844e3d8464020015f18b71c7f2c20ce340c619feed2663c3"}
Nov 03 15:03:21.347757 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.347724    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:03:21.553902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.553864    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:03:21.759408 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.759385    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:21.945962 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.945933    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca\") pod \"f23e89c6-7340-442f-9e94-945e784275f4\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") "
Nov 03 15:03:21.946161 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.946143    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca\") pod \"f23e89c6-7340-442f-9e94-945e784275f4\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") "
Nov 03 15:03:21.946257 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.946204    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x\" (UniqueName: \"kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x\") pod \"f23e89c6-7340-442f-9e94-945e784275f4\" (UID: \"f23e89c6-7340-442f-9e94-945e784275f4\") "
Nov 03 15:03:21.946497 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.946465    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca" pod "f23e89c6-7340-442f-9e94-945e784275f4" (UID: "f23e89c6-7340-442f-9e94-945e784275f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:03:21.946643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.946538    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:21.948015 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.947992    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca" pod "f23e89c6-7340-442f-9e94-945e784275f4" (UID: "f23e89c6-7340-442f-9e94-945e784275f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:03:21.948222 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:21.948207    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x" pod "f23e89c6-7340-442f-9e94-945e784275f4" (UID: "f23e89c6-7340-442f-9e94-945e784275f4"). InnerVolumeSpecName "kube-api-access-smc7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:03:22.047366 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:22.047332    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23e89c6-7340-442f-9e94-945e784275f4-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:22.047366 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:22.047361    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smc7x\" (UniqueName: \"kubernetes.io/projected/f23e89c6-7340-442f-9e94-945e784275f4-kube-api-access-smc7x\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:22.633542 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:22.633515    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"
Nov 03 15:03:22.633542 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:22.633531    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb" event={"ID":"f23e89c6-7340-442f-9e94-945e784275f4","Type":"ContainerDied","Data":"c72acfee6a2596430591739b313c05979ef76398784130d4082a20094263f28d"}
Nov 03 15:03:22.633769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:22.633573    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c72acfee6a2596430591739b313c05979ef76398784130d4082a20094263f28d"
Nov 03 15:03:25.827076 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:25.827043    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:03:27.985247 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:03:27.985200    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:03:28.815718 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.815684    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"]
Nov 03 15:03:28.816135 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.816119    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f23e89c6-7340-442f-9e94-945e784275f4" containerName="revoke-keys"
Nov 03 15:03:28.816186 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.816137    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23e89c6-7340-442f-9e94-945e784275f4" containerName="revoke-keys"
Nov 03 15:03:28.816275 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.816265    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23e89c6-7340-442f-9e94-945e784275f4" containerName="revoke-keys"
Nov 03 15:03:28.819930 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.819910    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:28.839439 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.839413    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"]
Nov 03 15:03:28.848073 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.848051    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:03:28.848073 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.848055    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:03:28.848273 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.848091    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:03:28.848273 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.848113    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:03:28.905674 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.905644    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:28.905835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.905690    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:28.905889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:28.905828    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84n5f\" (UniqueName: \"kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.007060 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.007031    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84n5f\" (UniqueName: \"kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.007455 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.007101    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.007455 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.007130    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.007785 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.007763    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.009449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.009425    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.027963 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.027940    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84n5f\" (UniqueName: \"kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f\") pod \"maas-api-revoke-sub-5a29efb3-xqq9p\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.132999 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.132942    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:29.431216 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.431190    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"]
Nov 03 15:03:29.432906 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:03:29.432872    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb95c23_d306_4019_b71c_514a13971cb2.slice/crio-1112dff1867f46c721eb38942272f2dfde3e7b453eb61ed2d079326ee85ee7d7 WatchSource:0}: Error finding container 1112dff1867f46c721eb38942272f2dfde3e7b453eb61ed2d079326ee85ee7d7: Status 404 returned error can't find the container with id 1112dff1867f46c721eb38942272f2dfde3e7b453eb61ed2d079326ee85ee7d7
Nov 03 15:03:29.671430 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.671401    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p" event={"ID":"6bb95c23-d306-4019-b71c-514a13971cb2","Type":"ContainerStarted","Data":"1112dff1867f46c721eb38942272f2dfde3e7b453eb61ed2d079326ee85ee7d7"}
Nov 03 15:03:29.700587 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.700526    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:03:29.709264 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.709223    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:03:29.918651 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:29.918620    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:03:30.682793 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:30.682753    6481 generic.go:356] "Generic (PLEG): container finished" podID="6bb95c23-d306-4019-b71c-514a13971cb2" containerID="6869c7e9e2b0f2cd83bd5f41870a10de49a1700c72d5c6917f4293a0cb34a733" exitCode=0
Nov 03 15:03:30.683353 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:30.682854    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p" event={"ID":"6bb95c23-d306-4019-b71c-514a13971cb2","Type":"ContainerDied","Data":"6869c7e9e2b0f2cd83bd5f41870a10de49a1700c72d5c6917f4293a0cb34a733"}
Nov 03 15:03:32.571310 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.571284    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:32.637571 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.637523    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca\") pod \"6bb95c23-d306-4019-b71c-514a13971cb2\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") "
Nov 03 15:03:32.637890 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.637854    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca\") pod \"6bb95c23-d306-4019-b71c-514a13971cb2\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") "
Nov 03 15:03:32.638102 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.638084    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f\" (UniqueName: \"kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f\") pod \"6bb95c23-d306-4019-b71c-514a13971cb2\" (UID: \"6bb95c23-d306-4019-b71c-514a13971cb2\") "
Nov 03 15:03:32.640751 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.640724    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca" pod "6bb95c23-d306-4019-b71c-514a13971cb2" (UID: "6bb95c23-d306-4019-b71c-514a13971cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:03:32.713418 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.713354    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca" pod "6bb95c23-d306-4019-b71c-514a13971cb2" (UID: "6bb95c23-d306-4019-b71c-514a13971cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:03:32.906392 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.838449    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f" pod "6bb95c23-d306-4019-b71c-514a13971cb2" (UID: "6bb95c23-d306-4019-b71c-514a13971cb2"). InnerVolumeSpecName "kube-api-access-84n5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:03:32.906392 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.872129    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:32.906392 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.872289    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bb95c23-d306-4019-b71c-514a13971cb2-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:32.906392 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.872315    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84n5f\" (UniqueName: \"kubernetes.io/projected/6bb95c23-d306-4019-b71c-514a13971cb2-kube-api-access-84n5f\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:32.983129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.983088    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p" event={"ID":"6bb95c23-d306-4019-b71c-514a13971cb2","Type":"ContainerDied","Data":"1112dff1867f46c721eb38942272f2dfde3e7b453eb61ed2d079326ee85ee7d7"}
Nov 03 15:03:32.983470 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.983448    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1112dff1867f46c721eb38942272f2dfde3e7b453eb61ed2d079326ee85ee7d7"
Nov 03 15:03:32.983708 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:32.983692    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"
Nov 03 15:03:33.233253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:33.229368    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:03:33.233253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:33.229448    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:03:33.233253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:33.229921    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:03:33.233253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:33.229959    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:03:33.434407 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:33.434351    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:03:33.434783 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:33.434748    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:03:40.274102 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:40.274067    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:03:44.746631 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.746578    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"]
Nov 03 15:03:44.758531 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.758501    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p2stb"]
Nov 03 15:03:44.800618 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.800591    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"]
Nov 03 15:03:44.801027 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.801014    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6bb95c23-d306-4019-b71c-514a13971cb2" containerName="revoke-keys"
Nov 03 15:03:44.801073 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.801029    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bb95c23-d306-4019-b71c-514a13971cb2" containerName="revoke-keys"
Nov 03 15:03:44.801152 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.801116    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bb95c23-d306-4019-b71c-514a13971cb2" containerName="revoke-keys"
Nov 03 15:03:44.804268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.804252    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:44.819682 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.819661    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:03:44.819793 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.819785    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:03:44.820217 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.820202    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:03:44.820376 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.820363    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:03:44.832813 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.832789    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"]
Nov 03 15:03:44.933435 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.933403    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:44.933601 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.933564    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4w7\" (UniqueName: \"kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:44.933670 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:44.933601    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.034357 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.034279    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4w7\" (UniqueName: \"kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.034357 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.034314    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.034546 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.034376    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.034978 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.034956    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.036656 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.036636    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.054829 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.054807    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4w7\" (UniqueName: \"kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7\") pod \"maas-api-revoke-sub-3d4156f1-kbrxt\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.117769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.117740    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:45.416215 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:45.416188    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"]
Nov 03 15:03:45.417368 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:03:45.417341    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1b9e85_7cae_4fc1_b808_d33a7a50b1b3.slice/crio-b59051e5a0b1d464d644d6f83cce4328d26c89a59895dac07834fc5984db02d9 WatchSource:0}: Error finding container b59051e5a0b1d464d644d6f83cce4328d26c89a59895dac07834fc5984db02d9: Status 404 returned error can't find the container with id b59051e5a0b1d464d644d6f83cce4328d26c89a59895dac07834fc5984db02d9
Nov 03 15:03:46.055947 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:46.055863    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt" event={"ID":"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3","Type":"ContainerStarted","Data":"aa26ade8f39ce3acb709b0b962ad5a5e24b704e09720e7e7d4c8173d817ec814"}
Nov 03 15:03:46.056324 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:46.055959    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt" event={"ID":"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3","Type":"ContainerStarted","Data":"b59051e5a0b1d464d644d6f83cce4328d26c89a59895dac07834fc5984db02d9"}
Nov 03 15:03:46.065683 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:46.065658    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e" path="/var/lib/kubelet/pods/1a0d1e6e-f1c5-47cb-b618-8a1cc419dc7e/volumes"
Nov 03 15:03:46.098059 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:46.098015    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt" podStartSLOduration=2.098002984 podStartE2EDuration="2.098002984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:03:46.095486937 +0000 UTC m=+3104.145518120" watchObservedRunningTime="2026-11-03 15:03:46.098002984 +0000 UTC m=+3104.148034166"
Nov 03 15:03:47.063015 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:47.062980    6481 generic.go:356] "Generic (PLEG): container finished" podID="7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" containerID="aa26ade8f39ce3acb709b0b962ad5a5e24b704e09720e7e7d4c8173d817ec814" exitCode=0
Nov 03 15:03:47.063427 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:47.063057    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt" event={"ID":"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3","Type":"ContainerDied","Data":"aa26ade8f39ce3acb709b0b962ad5a5e24b704e09720e7e7d4c8173d817ec814"}
Nov 03 15:03:48.208943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.208918    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:48.365155 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.365084    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca\") pod \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") "
Nov 03 15:03:48.365155 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.365130    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca\") pod \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") "
Nov 03 15:03:48.365396 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.365165    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7\" (UniqueName: \"kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7\") pod \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\" (UID: \"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3\") "
Nov 03 15:03:48.365643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.365621    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca" pod "7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" (UID: "7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:03:48.367153 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.367129    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca" pod "7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" (UID: "7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:03:48.367270 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.367213    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7" pod "7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" (UID: "7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3"). InnerVolumeSpecName "kube-api-access-nq4w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:03:48.466135 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.466108    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:48.466135 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.466133    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:48.466321 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:48.466142    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq4w7\" (UniqueName: \"kubernetes.io/projected/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3-kube-api-access-nq4w7\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:03:49.074428 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:49.074393    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt" event={"ID":"7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3","Type":"ContainerDied","Data":"b59051e5a0b1d464d644d6f83cce4328d26c89a59895dac07834fc5984db02d9"}
Nov 03 15:03:49.074614 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:49.074441    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b59051e5a0b1d464d644d6f83cce4328d26c89a59895dac07834fc5984db02d9"
Nov 03 15:03:49.074614 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:49.074405    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"
Nov 03 15:03:56.979412 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:03:56.979380    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:03:58.078258 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:03:58.070988    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:04:00.100965 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:00.100929    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:00.545486 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:00.545438    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:00.766076 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:00.766045    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:02.502750 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:02.502717    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:04:02.582499 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:02.582465    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:04.568000 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:04.567969    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:04.781023 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:04.780989    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:06.320957 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.320923    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:06.405391 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.405359    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"]
Nov 03 15:04:06.405857 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.405843    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" containerName="revoke-keys"
Nov 03 15:04:06.405922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.405858    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" containerName="revoke-keys"
Nov 03 15:04:06.406003 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.405991    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" containerName="revoke-keys"
Nov 03 15:04:06.408769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.408753    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.424613 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.424595    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ecd9e5\""
Nov 03 15:04:06.435588 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.435564    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-745e72e8-dockercfg-z5cfc\""
Nov 03 15:04:06.439048 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.439024    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"]
Nov 03 15:04:06.505532 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.505499    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"]
Nov 03 15:04:06.509765 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.509739    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.521904 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.521870    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.522056 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.522012    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qf6\" (UniqueName: \"kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.522276 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.522255    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"]
Nov 03 15:04:06.529304 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.529277    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:06.622887 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.622816    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rr9\" (UniqueName: \"kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.622887 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.622861    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qf6\" (UniqueName: \"kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.623056 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.622952    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.623093 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.623068    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.623711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.623693    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.643408 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.643388    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qf6\" (UniqueName: \"kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.721347 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.721319    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:06.723726 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.723698    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.723841 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.723818    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rr9\" (UniqueName: \"kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.724401 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.724380    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.744085 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.744062    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rr9\" (UniqueName: \"kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:06.824086 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:06.824058    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:07.022566 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.022540    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"]
Nov 03 15:04:07.023995 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:04:07.023970    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6cf518_1a64_4a11_9f44_eb6ed2ff278f.slice/crio-957cadf196b42375f20b4ef8d62c58fd028585d399c10f18934b43f583553235 WatchSource:0}: Error finding container 957cadf196b42375f20b4ef8d62c58fd028585d399c10f18934b43f583553235: Status 404 returned error can't find the container with id 957cadf196b42375f20b4ef8d62c58fd028585d399c10f18934b43f583553235
Nov 03 15:04:07.123424 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.123378    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"]
Nov 03 15:04:07.124883 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:04:07.124861    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb38647a_f9f1_4bab_b2c7_6447f75b7a44.slice/crio-6bdd848443c5cdf9e1efb73f7ab634a458c4cb39cb9778798b2f8cd9517a66c9 WatchSource:0}: Error finding container 6bdd848443c5cdf9e1efb73f7ab634a458c4cb39cb9778798b2f8cd9517a66c9: Status 404 returned error can't find the container with id 6bdd848443c5cdf9e1efb73f7ab634a458c4cb39cb9778798b2f8cd9517a66c9
Nov 03 15:04:07.212112 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.212084    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" event={"ID":"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f","Type":"ContainerStarted","Data":"cec02cb0bbd71483ced0daae1a26216c4254acc47b10bb63acb24fd054aa5ff4"}
Nov 03 15:04:07.212251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.212143    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:07.212251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.212166    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" event={"ID":"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f","Type":"ContainerStarted","Data":"957cadf196b42375f20b4ef8d62c58fd028585d399c10f18934b43f583553235"}
Nov 03 15:04:07.213694 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.213669    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" event={"ID":"db38647a-f9f1-4bab-b2c7-6447f75b7a44","Type":"ContainerStarted","Data":"8466f86ab6873fdd67e9489d60df06424a518b44f444da2bf069618235147b64"}
Nov 03 15:04:07.213798 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.213717    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" event={"ID":"db38647a-f9f1-4bab-b2c7-6447f75b7a44","Type":"ContainerStarted","Data":"6bdd848443c5cdf9e1efb73f7ab634a458c4cb39cb9778798b2f8cd9517a66c9"}
Nov 03 15:04:07.213798 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.213742    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:07.253317 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.253280    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" podStartSLOduration=1.253265118 podStartE2EDuration="1.253265118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:04:07.252538355 +0000 UTC m=+3125.302569538" watchObservedRunningTime="2026-11-03 15:04:07.253265118 +0000 UTC m=+3125.303296302"
Nov 03 15:04:07.295259 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:07.295203    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" podStartSLOduration=1.295191904 podStartE2EDuration="1.295191904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:04:07.294399812 +0000 UTC m=+3125.344430995" watchObservedRunningTime="2026-11-03 15:04:07.295191904 +0000 UTC m=+3125.345223087"
Nov 03 15:04:08.305900 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:08.305862    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:10.015250 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:10.014606    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:04:10.172152 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:10.172116    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:12.103067 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:12.103030    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:14.104785 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:14.104748    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:16.139909 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:16.139879    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:18.090327 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:18.090299    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:18.221761 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:18.221729    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:04:18.221945 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:18.221795    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:04:19.356509 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:19.356475    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:19.956568 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:19.956530    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:20.177440 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:20.177406    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:21.791491 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.791455    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"]
Nov 03 15:04:21.796411 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.796383    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:21.823672 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.823652    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ecd9e5\"/\"default-dockercfg-s5n26\""
Nov 03 15:04:21.823846 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.823791    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ecd9e5\"/\"simulator-single-main-ecd9e5-kserve-self-signed-certs\""
Nov 03 15:04:21.824170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.824137    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ecd9e5\"/\"kube-root-ca.crt\""
Nov 03 15:04:21.824170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.824144    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ecd9e5\"/\"openshift-service-ca.crt\""
Nov 03 15:04:21.827597 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.827574    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"]
Nov 03 15:04:21.930124 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.930092    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:21.973584 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.973555    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wpv9\" (UniqueName: \"kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:21.973726 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.973599    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:21.973726 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.973623    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:21.973850 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.973830    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:21.973901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.973863    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:21.973901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:21.973898    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075186 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075098    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075186 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075146    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075186 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075173    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075533 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075253    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wpv9\" (UniqueName: \"kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075533 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075305    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075533 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075348    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075723 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075647    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075723 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075684    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.075842 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.075754    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.077759 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.077736    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.077925 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.077841    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.095968 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.095939    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wpv9\" (UniqueName: \"kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9\") pod \"simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") " pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.111489 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.111468    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:22.150055 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.150013    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:22.408832 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:22.408810    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"]
Nov 03 15:04:22.410123 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:04:22.410095    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod502e9743_b1b5_4912_b7ca_425b86bd6b33.slice/crio-0fdc460a3efa8040ad628294bd7b9008a2f7045a644762ee8c765258c5f96d44 WatchSource:0}: Error finding container 0fdc460a3efa8040ad628294bd7b9008a2f7045a644762ee8c765258c5f96d44: Status 404 returned error can't find the container with id 0fdc460a3efa8040ad628294bd7b9008a2f7045a644762ee8c765258c5f96d44
Nov 03 15:04:23.306309 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:23.306268    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p" event={"ID":"502e9743-b1b5-4912-b7ca-425b86bd6b33","Type":"ContainerStarted","Data":"6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc"}
Nov 03 15:04:23.306794 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:23.306321    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p" event={"ID":"502e9743-b1b5-4912-b7ca-425b86bd6b33","Type":"ContainerStarted","Data":"0fdc460a3efa8040ad628294bd7b9008a2f7045a644762ee8c765258c5f96d44"}
Nov 03 15:04:23.346516 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:23.346464    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p" podStartSLOduration=2.34645203 podStartE2EDuration="2.34645203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:04:23.346436529 +0000 UTC m=+3141.396467711" watchObservedRunningTime="2026-11-03 15:04:23.34645203 +0000 UTC m=+3141.396483210"
Nov 03 15:04:23.919321 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:23.919288    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:24.112255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:24.112073    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:24.181323 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:24.181170    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:04:25.922977 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:25.922941    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:27.791289 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:27.791249    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:27.984943 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:04:27.984911    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:04:29.713322 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:29.713288    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:31.598907 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:31.598871    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:32.112078 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:32.112036    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:32.124351 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:32.124324    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:32.375712 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:32.375618    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:04:33.039797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:33.039764    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"]
Nov 03 15:04:33.051211 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:33.051185    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-s4ghv"]
Nov 03 15:04:33.495087 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:33.495031    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:34.065592 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:34.065559    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e21dfd9-823f-43fe-8694-79afe874668d" path="/var/lib/kubelet/pods/9e21dfd9-823f-43fe-8694-79afe874668d/volumes"
Nov 03 15:04:35.396573 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:35.396539    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:37.288960 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:37.288923    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:39.113779 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:39.113745    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:41.095681 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:41.095642    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:04:41.196591 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:41.196560    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:43.207575 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:43.207480    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:45.094160 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:45.094128    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:47.017426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:47.017390    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:49.004027 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:49.003987    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:50.986045 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:50.986014    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:52.897797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:52.897749    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:53.596313 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.596279    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"]
Nov 03 15:04:53.601440 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.601418    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.621787 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.621757    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"]
Nov 03 15:04:53.629970 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.629949    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:04:53.630175 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.630021    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:04:53.630175 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.630108    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:04:53.630175 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.630115    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:04:53.712612 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.712581    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2sbq\" (UniqueName: \"kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.712775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.712623    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.712831 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.712774    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.813411 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.813377    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.813595 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.813475    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2sbq\" (UniqueName: \"kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.813595 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.813510    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.814013 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.813991    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.815800 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.815776    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.835068 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.835045    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2sbq\" (UniqueName: \"kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq\") pod \"maas-api-revoke-sub-038cdddc-scd2w\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:53.917571 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:53.917551    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:54.216440 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:54.216412    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"]
Nov 03 15:04:54.218000 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:04:54.217974    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29685cb_fac6_4196_bfec_a3c672530a4b.slice/crio-e35c4a85d80af4d73057a3bff8f74e4f23c99bf783413aa159da030294f73664 WatchSource:0}: Error finding container e35c4a85d80af4d73057a3bff8f74e4f23c99bf783413aa159da030294f73664: Status 404 returned error can't find the container with id e35c4a85d80af4d73057a3bff8f74e4f23c99bf783413aa159da030294f73664
Nov 03 15:04:54.486028 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:54.485951    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w" event={"ID":"d29685cb-fac6-4196-bfec-a3c672530a4b","Type":"ContainerStarted","Data":"e35c4a85d80af4d73057a3bff8f74e4f23c99bf783413aa159da030294f73664"}
Nov 03 15:04:54.798259 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:54.798143    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:55.492804 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:55.492767    6481 generic.go:356] "Generic (PLEG): container finished" podID="d29685cb-fac6-4196-bfec-a3c672530a4b" containerID="6308d8c2088872d042dfb1ca102c9d2a7291be806698d921b25cc1b0cc98acf7" exitCode=0
Nov 03 15:04:55.493192 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:55.492831    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w" event={"ID":"d29685cb-fac6-4196-bfec-a3c672530a4b","Type":"ContainerDied","Data":"6308d8c2088872d042dfb1ca102c9d2a7291be806698d921b25cc1b0cc98acf7"}
Nov 03 15:04:56.636326 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.636294    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:56.702268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.702179    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:56.746948 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.746908    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca\") pod \"d29685cb-fac6-4196-bfec-a3c672530a4b\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") "
Nov 03 15:04:56.747108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.747007    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq\" (UniqueName: \"kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq\") pod \"d29685cb-fac6-4196-bfec-a3c672530a4b\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") "
Nov 03 15:04:56.747108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.747041    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca\") pod \"d29685cb-fac6-4196-bfec-a3c672530a4b\" (UID: \"d29685cb-fac6-4196-bfec-a3c672530a4b\") "
Nov 03 15:04:56.747356 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.747333    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca" pod "d29685cb-fac6-4196-bfec-a3c672530a4b" (UID: "d29685cb-fac6-4196-bfec-a3c672530a4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:04:56.747548 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.747530    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:04:56.749148 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.749084    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca" pod "d29685cb-fac6-4196-bfec-a3c672530a4b" (UID: "d29685cb-fac6-4196-bfec-a3c672530a4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:04:56.749148 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.749141    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq" pod "d29685cb-fac6-4196-bfec-a3c672530a4b" (UID: "d29685cb-fac6-4196-bfec-a3c672530a4b"). InnerVolumeSpecName "kube-api-access-m2sbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:04:56.848902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.848876    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2sbq\" (UniqueName: \"kubernetes.io/projected/d29685cb-fac6-4196-bfec-a3c672530a4b-kube-api-access-m2sbq\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:04:56.848902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:56.848901    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d29685cb-fac6-4196-bfec-a3c672530a4b-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:04:57.505127 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:57.505097    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"
Nov 03 15:04:57.505329 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:57.505139    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w" event={"ID":"d29685cb-fac6-4196-bfec-a3c672530a4b","Type":"ContainerDied","Data":"e35c4a85d80af4d73057a3bff8f74e4f23c99bf783413aa159da030294f73664"}
Nov 03 15:04:57.505329 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:57.505220    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e35c4a85d80af4d73057a3bff8f74e4f23c99bf783413aa159da030294f73664"
Nov 03 15:04:57.985653 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:04:57.985620    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:04:58.626605 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:58.626571    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:04:59.054835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:59.054800    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"]
Nov 03 15:04:59.075591 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:04:59.075560    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d7041ca-7wgls"]
Nov 03 15:05:00.066099 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:00.066069    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0a289b-2ec6-4003-8632-8433a9b4e235" path="/var/lib/kubelet/pods/8c0a289b-2ec6-4003-8632-8433a9b4e235/volumes"
Nov 03 15:05:00.599626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:00.599600    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:02.507403 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:02.507369    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:02.745610 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:02.745576    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:04.394168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:04.394137    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:04.784018 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:04.783985    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:05.266962 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:05.266936    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:06.112065 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:06.112033    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:05:06.300500 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:06.300470    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:08.221960 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:08.221927    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:09.649207 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:09.649177    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"]
Nov 03 15:05:09.877069 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:09.877042    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:10.456835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:10.456804    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"]
Nov 03 15:05:10.502836 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:10.502804    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"]
Nov 03 15:05:11.263029 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:11.262993    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:11.481994 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:11.481939    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:11.909467 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:11.909437    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:13.244442 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:13.244411    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:14.472208 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:14.472166    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:05:15.458085 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:15.458034    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerName="payload-processing" containerID="cri-o://8466f86ab6873fdd67e9489d60df06424a518b44f444da2bf069618235147b64" gracePeriod=25
Nov 03 15:05:15.504062 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:15.504013    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerName="payload-pre-processing" containerID="cri-o://cec02cb0bbd71483ced0daae1a26216c4254acc47b10bb63acb24fd054aa5ff4" gracePeriod=25
Nov 03 15:05:18.220044 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:18.220004    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:05:18.220486 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:18.220020    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:05:20.122594 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:20.122565    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:20.350508 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:20.350477    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:24.668715 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:24.668663    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p" podUID="502e9743-b1b5-4912-b7ca-425b86bd6b33" containerName="main" containerID="cri-o://6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc" gracePeriod=45
Nov 03 15:05:24.924046 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:24.923989    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:05:25.037007 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.036971    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home\") pod \"502e9743-b1b5-4912-b7ca-425b86bd6b33\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") "
Nov 03 15:05:25.037179 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037055    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache\") pod \"502e9743-b1b5-4912-b7ca-425b86bd6b33\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") "
Nov 03 15:05:25.037179 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037122    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9\" (UniqueName: \"kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9\") pod \"502e9743-b1b5-4912-b7ca-425b86bd6b33\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") "
Nov 03 15:05:25.037179 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037170    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir\") pod \"502e9743-b1b5-4912-b7ca-425b86bd6b33\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") "
Nov 03 15:05:25.037388 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037205    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm\") pod \"502e9743-b1b5-4912-b7ca-425b86bd6b33\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") "
Nov 03 15:05:25.037388 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037277    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home" pod "502e9743-b1b5-4912-b7ca-425b86bd6b33" (UID: "502e9743-b1b5-4912-b7ca-425b86bd6b33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:05:25.037388 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037305    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache" pod "502e9743-b1b5-4912-b7ca-425b86bd6b33" (UID: "502e9743-b1b5-4912-b7ca-425b86bd6b33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:05:25.037388 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037346    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs\" (UniqueName: \"kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs\") pod \"502e9743-b1b5-4912-b7ca-425b86bd6b33\" (UID: \"502e9743-b1b5-4912-b7ca-425b86bd6b33\") "
Nov 03 15:05:25.037605 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037451    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir" pod "502e9743-b1b5-4912-b7ca-425b86bd6b33" (UID: "502e9743-b1b5-4912-b7ca-425b86bd6b33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:05:25.037797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037766    6481 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-tmp-dir\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:25.037797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037798    6481 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-home\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:25.038012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.037814    6481 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-model-cache\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:25.039415 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.039390    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs" pod "502e9743-b1b5-4912-b7ca-425b86bd6b33" (UID: "502e9743-b1b5-4912-b7ca-425b86bd6b33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:05:25.039518 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.039429    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm" pod "502e9743-b1b5-4912-b7ca-425b86bd6b33" (UID: "502e9743-b1b5-4912-b7ca-425b86bd6b33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:05:25.039518 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.039454    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9" pod "502e9743-b1b5-4912-b7ca-425b86bd6b33" (UID: "502e9743-b1b5-4912-b7ca-425b86bd6b33"). InnerVolumeSpecName "kube-api-access-4wpv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:05:25.138970 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.138935    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wpv9\" (UniqueName: \"kubernetes.io/projected/502e9743-b1b5-4912-b7ca-425b86bd6b33-kube-api-access-4wpv9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:25.138970 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.138966    6481 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/502e9743-b1b5-4912-b7ca-425b86bd6b33-dshm\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:25.138970 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.138976    6481 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/502e9743-b1b5-4912-b7ca-425b86bd6b33-tls-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:25.666678 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.666642    6481 generic.go:356] "Generic (PLEG): container finished" podID="502e9743-b1b5-4912-b7ca-425b86bd6b33" containerID="6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc" exitCode=0
Nov 03 15:05:25.666678 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.666689    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"
Nov 03 15:05:25.666932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.666730    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p" event={"ID":"502e9743-b1b5-4912-b7ca-425b86bd6b33","Type":"ContainerDied","Data":"6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc"}
Nov 03 15:05:25.666932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.666776    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p" event={"ID":"502e9743-b1b5-4912-b7ca-425b86bd6b33","Type":"ContainerDied","Data":"0fdc460a3efa8040ad628294bd7b9008a2f7045a644762ee8c765258c5f96d44"}
Nov 03 15:05:25.666932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.666793    6481 scope.go:122] "RemoveContainer" containerID="6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc"
Nov 03 15:05:25.676540 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.676364    6481 scope.go:122] "RemoveContainer" containerID="6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc"
Nov 03 15:05:25.676788 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:05:25.676634    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc\": container with ID starting with 6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc not found: ID does not exist" containerID="6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc"
Nov 03 15:05:25.676788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.676660    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc"} err="failed to get container status \"6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc\": rpc error: code = NotFound desc = could not find container \"6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc\": container with ID starting with 6c1d15d63abcd810539dc75aeced94422f8752bfc87a1cad6d6aad9140ce9dcc not found: ID does not exist"
Nov 03 15:05:25.721056 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.721020    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"]
Nov 03 15:05:25.726994 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:25.726967    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ecd9e5/simulator-single-main-ecd9e5-kserve-7b7c44cc87-h6m4p"]
Nov 03 15:05:26.066568 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:26.066489    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502e9743-b1b5-4912-b7ca-425b86bd6b33" path="/var/lib/kubelet/pods/502e9743-b1b5-4912-b7ca-425b86bd6b33/volumes"
Nov 03 15:05:27.988468 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:05:27.988434    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:05:28.220108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:28.220068    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:05:28.220310 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:28.220071    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 15:05:32.043176 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:32.043142    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"]
Nov 03 15:05:32.054637 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:32.054612    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-zqfbt"]
Nov 03 15:05:32.065624 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:32.065596    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43e5756-22cf-43d4-8dff-2c664efa9075" path="/var/lib/kubelet/pods/d43e5756-22cf-43d4-8dff-2c664efa9075/volumes"
Nov 03 15:05:35.724657 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.724579    6481 generic.go:356] "Generic (PLEG): container finished" podID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerID="8466f86ab6873fdd67e9489d60df06424a518b44f444da2bf069618235147b64" exitCode=0
Nov 03 15:05:35.725114 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.724655    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" event={"ID":"db38647a-f9f1-4bab-b2c7-6447f75b7a44","Type":"ContainerDied","Data":"8466f86ab6873fdd67e9489d60df06424a518b44f444da2bf069618235147b64"}
Nov 03 15:05:35.726702 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.726680    6481 generic.go:356] "Generic (PLEG): container finished" podID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerID="cec02cb0bbd71483ced0daae1a26216c4254acc47b10bb63acb24fd054aa5ff4" exitCode=0
Nov 03 15:05:35.726806 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.726754    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" event={"ID":"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f","Type":"ContainerDied","Data":"cec02cb0bbd71483ced0daae1a26216c4254acc47b10bb63acb24fd054aa5ff4"}
Nov 03 15:05:35.755331 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.755308    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:05:35.779494 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.779466    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:05:35.842619 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.842592    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume\") pod \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") "
Nov 03 15:05:35.842788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.842685    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6\" (UniqueName: \"kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6\") pod \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") "
Nov 03 15:05:35.842788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.842781    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9\" (UniqueName: \"kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9\") pod \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\" (UID: \"db38647a-f9f1-4bab-b2c7-6447f75b7a44\") "
Nov 03 15:05:35.842915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.842822    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume\") pod \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\" (UID: \"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f\") "
Nov 03 15:05:35.843014 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.842986    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume" pod "db38647a-f9f1-4bab-b2c7-6447f75b7a44" (UID: "db38647a-f9f1-4bab-b2c7-6447f75b7a44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:05:35.843300 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.843273    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume" pod "9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" (UID: "9a6cf518-1a64-4a11-9f44-eb6ed2ff278f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:05:35.843425 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.843335    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db38647a-f9f1-4bab-b2c7-6447f75b7a44-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:35.844974 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.844950    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9" pod "db38647a-f9f1-4bab-b2c7-6447f75b7a44" (UID: "db38647a-f9f1-4bab-b2c7-6447f75b7a44"). InnerVolumeSpecName "kube-api-access-p7rr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:05:35.845109 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.845051    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6" pod "9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" (UID: "9a6cf518-1a64-4a11-9f44-eb6ed2ff278f"). InnerVolumeSpecName "kube-api-access-n8qf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:05:35.943873 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.943840    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rr9\" (UniqueName: \"kubernetes.io/projected/db38647a-f9f1-4bab-b2c7-6447f75b7a44-kube-api-access-p7rr9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:35.943873 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.943867    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:35.943873 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:35.943876    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8qf6\" (UniqueName: \"kubernetes.io/projected/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f-kube-api-access-n8qf6\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:36.733099 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.733019    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"
Nov 03 15:05:36.733099 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.733048    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf" event={"ID":"9a6cf518-1a64-4a11-9f44-eb6ed2ff278f","Type":"ContainerDied","Data":"957cadf196b42375f20b4ef8d62c58fd028585d399c10f18934b43f583553235"}
Nov 03 15:05:36.733640 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.733139    6481 scope.go:122] "RemoveContainer" containerID="cec02cb0bbd71483ced0daae1a26216c4254acc47b10bb63acb24fd054aa5ff4"
Nov 03 15:05:36.734547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.734524    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w" event={"ID":"db38647a-f9f1-4bab-b2c7-6447f75b7a44","Type":"ContainerDied","Data":"6bdd848443c5cdf9e1efb73f7ab634a458c4cb39cb9778798b2f8cd9517a66c9"}
Nov 03 15:05:36.734615 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.734542    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"
Nov 03 15:05:36.742474 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.742453    6481 scope.go:122] "RemoveContainer" containerID="8466f86ab6873fdd67e9489d60df06424a518b44f444da2bf069618235147b64"
Nov 03 15:05:36.787355 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.787331    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"]
Nov 03 15:05:36.799003 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.798977    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bd4n55w"]
Nov 03 15:05:36.826051 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.826027    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"]
Nov 03 15:05:36.833222 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:36.833199    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95k55zf"]
Nov 03 15:05:38.066606 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:38.066564    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" path="/var/lib/kubelet/pods/9a6cf518-1a64-4a11-9f44-eb6ed2ff278f/volumes"
Nov 03 15:05:38.067142 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:38.067120    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" path="/var/lib/kubelet/pods/db38647a-f9f1-4bab-b2c7-6447f75b7a44/volumes"
Nov 03 15:05:41.211971 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:41.211940    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:41.433563 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:41.433514    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:42.555300 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:42.555265    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:42.788830 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:42.788798    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:44.154835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.154803    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:44.362781 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.362749    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"]
Nov 03 15:05:44.363164 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363150    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d29685cb-fac6-4196-bfec-a3c672530a4b" containerName="revoke-keys"
Nov 03 15:05:44.363215 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363165    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d29685cb-fac6-4196-bfec-a3c672530a4b" containerName="revoke-keys"
Nov 03 15:05:44.363215 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363179    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerName="payload-pre-processing"
Nov 03 15:05:44.363215 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363186    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerName="payload-pre-processing"
Nov 03 15:05:44.363215 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363213    6481 cpu_manager.go:414] "containerMap: removing container" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerName="payload-processing"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363218    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerName="payload-processing"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363244    6481 cpu_manager.go:414] "containerMap: removing container" podUID="502e9743-b1b5-4912-b7ca-425b86bd6b33" containerName="main"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363249    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502e9743-b1b5-4912-b7ca-425b86bd6b33" containerName="main"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363308    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="db38647a-f9f1-4bab-b2c7-6447f75b7a44" containerName="payload-processing"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363317    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="502e9743-b1b5-4912-b7ca-425b86bd6b33" containerName="main"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363325    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d29685cb-fac6-4196-bfec-a3c672530a4b" containerName="revoke-keys"
Nov 03 15:05:44.363372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.363330    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a6cf518-1a64-4a11-9f44-eb6ed2ff278f" containerName="payload-pre-processing"
Nov 03 15:05:44.368098 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.368079    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.386881 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.386855    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:44.389089 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.389067    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"]
Nov 03 15:05:44.397825 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.397785    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ecd9e5\""
Nov 03 15:05:44.398219 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.398187    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-745e72e8-dockercfg-5z7kc\""
Nov 03 15:05:44.458425 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.458390    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"]
Nov 03 15:05:44.462178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.462161    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.494265 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.490772    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"]
Nov 03 15:05:44.526589 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.526552    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22r9\" (UniqueName: \"kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.526768 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.526616    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.628106 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.628072    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwwg\" (UniqueName: \"kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.628288 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.628176    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22r9\" (UniqueName: \"kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.628288 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.628220    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.628449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.628427    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.628949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.628931    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.648925 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.648900    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:44.649764 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.649744    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22r9\" (UniqueName: \"kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9\") pod \"payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.680581 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.680550    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:44.729932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.729740    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwwg\" (UniqueName: \"kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.729932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.729851    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.731439 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.731411    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.755413 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.755382    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwwg\" (UniqueName: \"kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg\") pod \"payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.784108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.784070    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:44.982510 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:44.982458    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"]
Nov 03 15:05:44.983877 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:05:44.983851    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea564064_13f9_40fa_b853_4a4ab4bfac6a.slice/crio-8ae0a7623ddf7735439c0f39d1dc76f26e9032613983965ba7dcbf34e6bf4477 WatchSource:0}: Error finding container 8ae0a7623ddf7735439c0f39d1dc76f26e9032613983965ba7dcbf34e6bf4477: Status 404 returned error can't find the container with id 8ae0a7623ddf7735439c0f39d1dc76f26e9032613983965ba7dcbf34e6bf4477
Nov 03 15:05:45.096147 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.096124    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"]
Nov 03 15:05:45.097466 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:05:45.097440    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1a6cd3_4772_4d2c_9250_10ccdc1fd556.slice/crio-2cb70a6f34f6cc10ee33057461062138477ecc08813e4a0eabd08356c4078ab4 WatchSource:0}: Error finding container 2cb70a6f34f6cc10ee33057461062138477ecc08813e4a0eabd08356c4078ab4: Status 404 returned error can't find the container with id 2cb70a6f34f6cc10ee33057461062138477ecc08813e4a0eabd08356c4078ab4
Nov 03 15:05:45.788576 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.788536    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4" event={"ID":"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556","Type":"ContainerStarted","Data":"73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875"}
Nov 03 15:05:45.789163 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.788606    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4" event={"ID":"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556","Type":"ContainerStarted","Data":"2cb70a6f34f6cc10ee33057461062138477ecc08813e4a0eabd08356c4078ab4"}
Nov 03 15:05:45.789163 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.788640    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:45.790142 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.790116    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h" event={"ID":"ea564064-13f9-40fa-b853-4a4ab4bfac6a","Type":"ContainerStarted","Data":"a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d"}
Nov 03 15:05:45.790256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.790161    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h" event={"ID":"ea564064-13f9-40fa-b853-4a4ab4bfac6a","Type":"ContainerStarted","Data":"8ae0a7623ddf7735439c0f39d1dc76f26e9032613983965ba7dcbf34e6bf4477"}
Nov 03 15:05:45.790256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.790188    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:45.832000 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.831948    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4" podStartSLOduration=1.8319341900000001 podStartE2EDuration="1.83193419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:05:45.83136593 +0000 UTC m=+3223.881397114" watchObservedRunningTime="2026-11-03 15:05:45.83193419 +0000 UTC m=+3223.881965373"
Nov 03 15:05:45.875204 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:45.875124    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h" podStartSLOduration=1.875115429 podStartE2EDuration="1.875115429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:05:45.874679563 +0000 UTC m=+3223.924710746" watchObservedRunningTime="2026-11-03 15:05:45.875115429 +0000 UTC m=+3223.925146612"
Nov 03 15:05:46.308965 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:46.308932    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:48.061449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:48.061415    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:48.296511 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:48.296472    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:48.318528 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:48.318441    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:05:49.665827 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:49.665792    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:50.212094 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:50.212060    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:52.101532 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:52.101500    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:53.635544 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.635511    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"]
Nov 03 15:05:53.639448 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.639428    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.660397 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.660375    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"]
Nov 03 15:05:53.669117 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.669093    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:05:53.669286 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.669266    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:05:53.669393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.669158    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:05:53.669597 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.669577    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:05:53.722505 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.722480    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.722641 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.722532    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lppt\" (UniqueName: \"kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.722684 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.722649    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.823999 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.823966    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lppt\" (UniqueName: \"kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.824129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.824036    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.824129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.824092    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.824676 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.824644    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.826321 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.826302    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.854314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.854293    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lppt\" (UniqueName: \"kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:53.953009 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:53.952989    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:54.044111 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:54.044058    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:54.260706 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:54.260682    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"]
Nov 03 15:05:54.263244 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:05:54.263206    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa70dfe_b557_4147_bfde_9ddb0b549096.slice/crio-579b6a181adc93c88b8af27ff44913093f57cefd31322c47434467d9a54058ee WatchSource:0}: Error finding container 579b6a181adc93c88b8af27ff44913093f57cefd31322c47434467d9a54058ee: Status 404 returned error can't find the container with id 579b6a181adc93c88b8af27ff44913093f57cefd31322c47434467d9a54058ee
Nov 03 15:05:54.844049 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:54.843965    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw" event={"ID":"aaa70dfe-b557-4147-bfde-9ddb0b549096","Type":"ContainerStarted","Data":"e9b7aa718f2eb52df40f804985b01d327da76c1199b69cc1cb2eaf9b6080413e"}
Nov 03 15:05:54.844049 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:54.844026    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw" event={"ID":"aaa70dfe-b557-4147-bfde-9ddb0b549096","Type":"ContainerStarted","Data":"579b6a181adc93c88b8af27ff44913093f57cefd31322c47434467d9a54058ee"}
Nov 03 15:05:54.859045 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:54.859015    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:05:54.888709 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:54.888666    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw" podStartSLOduration=1.888652992 podStartE2EDuration="1.888652992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:05:54.888458448 +0000 UTC m=+3232.938489632" watchObservedRunningTime="2026-11-03 15:05:54.888652992 +0000 UTC m=+3232.938684174"
Nov 03 15:05:55.851366 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:55.851329    6481 generic.go:356] "Generic (PLEG): container finished" podID="aaa70dfe-b557-4147-bfde-9ddb0b549096" containerID="e9b7aa718f2eb52df40f804985b01d327da76c1199b69cc1cb2eaf9b6080413e" exitCode=0
Nov 03 15:05:55.851803 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:55.851373    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw" event={"ID":"aaa70dfe-b557-4147-bfde-9ddb0b549096","Type":"ContainerDied","Data":"e9b7aa718f2eb52df40f804985b01d327da76c1199b69cc1cb2eaf9b6080413e"}
Nov 03 15:05:56.127081 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:56.127004    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:56.798949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:56.798924    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:05:56.799137 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:56.798987    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:05:57.007134 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.007111    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:57.055525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.055445    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt\" (UniqueName: \"kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt\") pod \"aaa70dfe-b557-4147-bfde-9ddb0b549096\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") "
Nov 03 15:05:57.055688 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.055556    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca\") pod \"aaa70dfe-b557-4147-bfde-9ddb0b549096\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") "
Nov 03 15:05:57.055688 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.055637    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca\") pod \"aaa70dfe-b557-4147-bfde-9ddb0b549096\" (UID: \"aaa70dfe-b557-4147-bfde-9ddb0b549096\") "
Nov 03 15:05:57.056252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.056208    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca" pod "aaa70dfe-b557-4147-bfde-9ddb0b549096" (UID: "aaa70dfe-b557-4147-bfde-9ddb0b549096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:05:57.059668 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.059640    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt" pod "aaa70dfe-b557-4147-bfde-9ddb0b549096" (UID: "aaa70dfe-b557-4147-bfde-9ddb0b549096"). InnerVolumeSpecName "kube-api-access-6lppt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:05:57.065611 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.065579    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca" pod "aaa70dfe-b557-4147-bfde-9ddb0b549096" (UID: "aaa70dfe-b557-4147-bfde-9ddb0b549096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:05:57.134194 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.134152    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:57.156569 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.156546    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:57.156704 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.156586    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lppt\" (UniqueName: \"kubernetes.io/projected/aaa70dfe-b557-4147-bfde-9ddb0b549096-kube-api-access-6lppt\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:57.156704 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.156596    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaa70dfe-b557-4147-bfde-9ddb0b549096-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:05:57.375607 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.375526    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:57.873843 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.873784    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw" event={"ID":"aaa70dfe-b557-4147-bfde-9ddb0b549096","Type":"ContainerDied","Data":"579b6a181adc93c88b8af27ff44913093f57cefd31322c47434467d9a54058ee"}
Nov 03 15:05:57.874075 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.873928    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="579b6a181adc93c88b8af27ff44913093f57cefd31322c47434467d9a54058ee"
Nov 03 15:05:57.874075 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.873945    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"
Nov 03 15:05:57.910656 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:57.910628    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:05:57.985435 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:05:57.985401    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:05:59.809108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:05:59.809073    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:01.044142 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:01.044111    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"]
Nov 03 15:06:01.055260 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:01.055217    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-8l9fm"]
Nov 03 15:06:01.695266 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:01.695225    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:02.066704 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:02.066629    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19124cf0-85ec-41ad-941b-0b793f546267" path="/var/lib/kubelet/pods/19124cf0-85ec-41ad-941b-0b793f546267/volumes"
Nov 03 15:06:02.829043 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:02.829009    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:03.694047 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:03.694004    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:04.156505 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:04.156425    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"]
Nov 03 15:06:04.190393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:04.190365    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"]
Nov 03 15:06:04.410856 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:04.410830    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:04.639403 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:04.639374    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:05.594102 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:05.594065    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:07.255877 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:07.255846    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:07.897720 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:07.897691    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:07.986980 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:07.986945    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"]
Nov 03 15:06:07.997813 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:07.997786    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ecd9e5-dpslw"]
Nov 03 15:06:08.065436 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:08.065405    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaa70dfe-b557-4147-bfde-9ddb0b549096" path="/var/lib/kubelet/pods/aaa70dfe-b557-4147-bfde-9ddb0b549096/volumes"
Nov 03 15:06:08.124958 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:08.124931    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:08.459477 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:08.459439    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:06:08.934650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:08.934620    6481 scope.go:122] "RemoveContainer" containerID="8b5d146db50c4f57e9b7dbb43d2d0e12198dd41bc35798800e0211288d03b150"
Nov 03 15:06:08.943371 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:08.943351    6481 scope.go:122] "RemoveContainer" containerID="f3d66da3c0f73738345cbf48d5c2a90dad555c9f3c10c0ed2d8b86a65750d6bc"
Nov 03 15:06:09.157170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.157117    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4" podUID="9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" containerName="payload-processing" containerID="cri-o://73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875" gracePeriod=25
Nov 03 15:06:09.190840 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.190784    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h" podUID="ea564064-13f9-40fa-b853-4a4ab4bfac6a" containerName="payload-pre-processing" containerID="cri-o://a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d" gracePeriod=25
Nov 03 15:06:09.465636 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.465372    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:06:09.493443 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.493420    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:06:09.573142 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.573111    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume\") pod \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") "
Nov 03 15:06:09.573336 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.573152    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg\" (UniqueName: \"kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg\") pod \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") "
Nov 03 15:06:09.573408 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.573347    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9\" (UniqueName: \"kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9\") pod \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\" (UID: \"ea564064-13f9-40fa-b853-4a4ab4bfac6a\") "
Nov 03 15:06:09.573474 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.573453    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume\") pod \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\" (UID: \"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556\") "
Nov 03 15:06:09.573726 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.573619    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume" pod "ea564064-13f9-40fa-b853-4a4ab4bfac6a" (UID: "ea564064-13f9-40fa-b853-4a4ab4bfac6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:06:09.573804 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.573783    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume" pod "9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" (UID: "9e1a6cd3-4772-4d2c-9250-10ccdc1fd556"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:06:09.575564 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.575541    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg" pod "9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" (UID: "9e1a6cd3-4772-4d2c-9250-10ccdc1fd556"). InnerVolumeSpecName "kube-api-access-gzwwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:06:09.575646 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.575631    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9" pod "ea564064-13f9-40fa-b853-4a4ab4bfac6a" (UID: "ea564064-13f9-40fa-b853-4a4ab4bfac6a"). InnerVolumeSpecName "kube-api-access-b22r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:06:09.674852 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.674821    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:09.674852 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.674851    6481 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea564064-13f9-40fa-b853-4a4ab4bfac6a-plugins-config-volume\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:09.675076 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.674864    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzwwg\" (UniqueName: \"kubernetes.io/projected/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556-kube-api-access-gzwwg\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:09.675076 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.674876    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b22r9\" (UniqueName: \"kubernetes.io/projected/ea564064-13f9-40fa-b853-4a4ab4bfac6a-kube-api-access-b22r9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:09.686286 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.686251    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"]
Nov 03 15:06:09.691160 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.691138    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kbrxt"]
Nov 03 15:06:09.735467 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735406    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"]
Nov 03 15:06:09.735845 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735833    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" containerName="payload-processing"
Nov 03 15:06:09.735889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735846    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" containerName="payload-processing"
Nov 03 15:06:09.735889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735864    6481 cpu_manager.go:414] "containerMap: removing container" podUID="aaa70dfe-b557-4147-bfde-9ddb0b549096" containerName="revoke-keys"
Nov 03 15:06:09.735889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735869    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaa70dfe-b557-4147-bfde-9ddb0b549096" containerName="revoke-keys"
Nov 03 15:06:09.735889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735880    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ea564064-13f9-40fa-b853-4a4ab4bfac6a" containerName="payload-pre-processing"
Nov 03 15:06:09.735889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735885    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea564064-13f9-40fa-b853-4a4ab4bfac6a" containerName="payload-pre-processing"
Nov 03 15:06:09.736051 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735958    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea564064-13f9-40fa-b853-4a4ab4bfac6a" containerName="payload-pre-processing"
Nov 03 15:06:09.736051 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735973    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" containerName="payload-processing"
Nov 03 15:06:09.736051 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.735980    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaa70dfe-b557-4147-bfde-9ddb0b549096" containerName="revoke-keys"
Nov 03 15:06:09.739423 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.739407    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.757908 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.757877    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:06:09.758086 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.758065    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:06:09.758196 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.758078    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:06:09.758196 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.758089    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:06:09.759877 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.759857    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"]
Nov 03 15:06:09.876306 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.876271    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.876499 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.876310    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.876499 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.876332    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5xk\" (UniqueName: \"kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.942139 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.942105    6481 generic.go:356] "Generic (PLEG): container finished" podID="9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" containerID="73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875" exitCode=0
Nov 03 15:06:09.942349 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.942153    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"
Nov 03 15:06:09.942349 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.942193    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4" event={"ID":"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556","Type":"ContainerDied","Data":"73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875"}
Nov 03 15:06:09.942349 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.942259    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4" event={"ID":"9e1a6cd3-4772-4d2c-9250-10ccdc1fd556","Type":"ContainerDied","Data":"2cb70a6f34f6cc10ee33057461062138477ecc08813e4a0eabd08356c4078ab4"}
Nov 03 15:06:09.942349 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.942295    6481 scope.go:122] "RemoveContainer" containerID="73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875"
Nov 03 15:06:09.943541 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.943522    6481 generic.go:356] "Generic (PLEG): container finished" podID="ea564064-13f9-40fa-b853-4a4ab4bfac6a" containerID="a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d" exitCode=0
Nov 03 15:06:09.943618 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.943590    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"
Nov 03 15:06:09.943686 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.943609    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h" event={"ID":"ea564064-13f9-40fa-b853-4a4ab4bfac6a","Type":"ContainerDied","Data":"a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d"}
Nov 03 15:06:09.943686 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.943665    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h" event={"ID":"ea564064-13f9-40fa-b853-4a4ab4bfac6a","Type":"ContainerDied","Data":"8ae0a7623ddf7735439c0f39d1dc76f26e9032613983965ba7dcbf34e6bf4477"}
Nov 03 15:06:09.951320 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.951304    6481 scope.go:122] "RemoveContainer" containerID="73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875"
Nov 03 15:06:09.951629 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:06:09.951593    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875\": container with ID starting with 73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875 not found: ID does not exist" containerID="73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875"
Nov 03 15:06:09.951728 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.951640    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875"} err="failed to get container status \"73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875\": rpc error: code = NotFound desc = could not find container \"73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875\": container with ID starting with 73863009657b52d3d1465b615f35d4a920be206161927286c91ef76fe1a26875 not found: ID does not exist"
Nov 03 15:06:09.951728 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.951663    6481 scope.go:122] "RemoveContainer" containerID="a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d"
Nov 03 15:06:09.960491 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.960476    6481 scope.go:122] "RemoveContainer" containerID="a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d"
Nov 03 15:06:09.960742 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:06:09.960724    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d\": container with ID starting with a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d not found: ID does not exist" containerID="a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d"
Nov 03 15:06:09.960790 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.960748    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d"} err="failed to get container status \"a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d\": rpc error: code = NotFound desc = could not find container \"a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d\": container with ID starting with a93bb328774a324be4b651a6a808df02592bc04d8001fbc5ed091ed139be198d not found: ID does not exist"
Nov 03 15:06:09.977715 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.977691    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.977828 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.977724    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.977828 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.977745    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5xk\" (UniqueName: \"kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.978451 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.978432    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.979922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.979903    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:09.995859 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.995735    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"]
Nov 03 15:06:09.999037 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.999015    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ecd9e5-5876c66bdx5jh4"]
Nov 03 15:06:09.999636 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:09.999614    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5xk\" (UniqueName: \"kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk\") pod \"maas-api-revoke-sub-3d4156f1-wjg86\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:10.035285 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.035251    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"]
Nov 03 15:06:10.042520 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.042493    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ecd9e5-5bc95xwt8h"]
Nov 03 15:06:10.052217 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.052197    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:10.066150 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.066122    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3" path="/var/lib/kubelet/pods/7b1b9e85-7cae-4fc1-b808-d33a7a50b1b3/volumes"
Nov 03 15:06:10.066625 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.066604    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1a6cd3-4772-4d2c-9250-10ccdc1fd556" path="/var/lib/kubelet/pods/9e1a6cd3-4772-4d2c-9250-10ccdc1fd556/volumes"
Nov 03 15:06:10.067009 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.066991    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea564064-13f9-40fa-b853-4a4ab4bfac6a" path="/var/lib/kubelet/pods/ea564064-13f9-40fa-b853-4a4ab4bfac6a/volumes"
Nov 03 15:06:10.356589 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.356558    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"]
Nov 03 15:06:10.356866 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:06:10.356844    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0274ec84_600c_4062_be83_dc32ddd769e0.slice/crio-fc4581b0407089832376d255a3f896cf8f7de5bcbad21a4f463fd30cfc396747 WatchSource:0}: Error finding container fc4581b0407089832376d255a3f896cf8f7de5bcbad21a4f463fd30cfc396747: Status 404 returned error can't find the container with id fc4581b0407089832376d255a3f896cf8f7de5bcbad21a4f463fd30cfc396747
Nov 03 15:06:10.377203 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.377177    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:06:10.950812 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.950778    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86" event={"ID":"0274ec84-600c-4062-be83-dc32ddd769e0","Type":"ContainerStarted","Data":"ff0532cb11434095857e9fa09b62e2796ef5b7888713d48e967e8d75f98c27e1"}
Nov 03 15:06:10.951254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.950829    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86" event={"ID":"0274ec84-600c-4062-be83-dc32ddd769e0","Type":"ContainerStarted","Data":"fc4581b0407089832376d255a3f896cf8f7de5bcbad21a4f463fd30cfc396747"}
Nov 03 15:06:10.995199 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:10.995154    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86" podStartSLOduration=1.9951403559999998 podStartE2EDuration="1.995140356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:06:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:06:10.993701219 +0000 UTC m=+3249.043732401" watchObservedRunningTime="2026-11-03 15:06:10.995140356 +0000 UTC m=+3249.045171555"
Nov 03 15:06:11.960635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:11.960604    6481 generic.go:356] "Generic (PLEG): container finished" podID="0274ec84-600c-4062-be83-dc32ddd769e0" containerID="ff0532cb11434095857e9fa09b62e2796ef5b7888713d48e967e8d75f98c27e1" exitCode=0
Nov 03 15:06:11.960635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:11.960638    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86" event={"ID":"0274ec84-600c-4062-be83-dc32ddd769e0","Type":"ContainerDied","Data":"ff0532cb11434095857e9fa09b62e2796ef5b7888713d48e967e8d75f98c27e1"}
Nov 03 15:06:12.053831 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:12.053799    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"]
Nov 03 15:06:12.057826 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:12.057802    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-5bhmc"]
Nov 03 15:06:12.065967 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:12.065931    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6da399-a62b-4519-a576-416d2d1ef2af" path="/var/lib/kubelet/pods/2d6da399-a62b-4519-a576-416d2d1ef2af/volumes"
Nov 03 15:06:13.105379 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.105355    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:13.211342 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.211307    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca\") pod \"0274ec84-600c-4062-be83-dc32ddd769e0\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") "
Nov 03 15:06:13.211529 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.211512    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca\") pod \"0274ec84-600c-4062-be83-dc32ddd769e0\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") "
Nov 03 15:06:13.211605 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.211564    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk\" (UniqueName: \"kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk\") pod \"0274ec84-600c-4062-be83-dc32ddd769e0\" (UID: \"0274ec84-600c-4062-be83-dc32ddd769e0\") "
Nov 03 15:06:13.211865 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.211839    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca" pod "0274ec84-600c-4062-be83-dc32ddd769e0" (UID: "0274ec84-600c-4062-be83-dc32ddd769e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:06:13.213332 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.213309    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca" pod "0274ec84-600c-4062-be83-dc32ddd769e0" (UID: "0274ec84-600c-4062-be83-dc32ddd769e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:06:13.213573 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.213547    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk" pod "0274ec84-600c-4062-be83-dc32ddd769e0" (UID: "0274ec84-600c-4062-be83-dc32ddd769e0"). InnerVolumeSpecName "kube-api-access-vr5xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:06:13.312722 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.312688    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:13.312722 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.312715    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5xk\" (UniqueName: \"kubernetes.io/projected/0274ec84-600c-4062-be83-dc32ddd769e0-kube-api-access-vr5xk\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:13.312722 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.312726    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0274ec84-600c-4062-be83-dc32ddd769e0-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:06:13.973174 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.973145    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"
Nov 03 15:06:13.973350 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.973186    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86" event={"ID":"0274ec84-600c-4062-be83-dc32ddd769e0","Type":"ContainerDied","Data":"fc4581b0407089832376d255a3f896cf8f7de5bcbad21a4f463fd30cfc396747"}
Nov 03 15:06:13.973350 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:13.973245    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc4581b0407089832376d255a3f896cf8f7de5bcbad21a4f463fd30cfc396747"
Nov 03 15:06:21.969430 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:21.969393    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:06:27.985843 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:06:27.985802    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:06:37.041578 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:37.041544    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"]
Nov 03 15:06:37.054413 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:37.054386    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-4rpkq"]
Nov 03 15:06:38.065747 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:38.065718    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="465161c8-91a0-4363-9d57-5b67d3946dca" path="/var/lib/kubelet/pods/465161c8-91a0-4363-9d57-5b67d3946dca/volumes"
Nov 03 15:06:53.051736 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:53.051686    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"]
Nov 03 15:06:53.063371 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:53.063346    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ff8kl"]
Nov 03 15:06:54.067303 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:06:54.067252    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1894d814-757b-4aff-944d-0faea157b49c" path="/var/lib/kubelet/pods/1894d814-757b-4aff-944d-0faea157b49c/volumes"
Nov 03 15:06:57.985764 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:06:57.985722    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:07:02.964847 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:02.964807    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:07:08.979789 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:08.979754    6481 scope.go:122] "RemoveContainer" containerID="be54b3ac833af126fb0192787774768d7f424011b717336773423938d7be758d"
Nov 03 15:07:08.988751 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:08.988729    6481 scope.go:122] "RemoveContainer" containerID="486e9bea99596333bbd0903b7e0a76bbf5715f67f95910480a61bd89b5cb5427"
Nov 03 15:07:08.996774 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:08.996759    6481 scope.go:122] "RemoveContainer" containerID="7e27910a2c7fe991fa33ab72527a231b367550e1fd5e3cb66a4d725d3b64a47c"
Nov 03 15:07:10.040716 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:10.040680    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"]
Nov 03 15:07:10.052182 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:10.052153    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-7q7w4"]
Nov 03 15:07:10.072091 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:10.072057    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e952a133-9bdb-4af5-8b16-38df266c53cd" path="/var/lib/kubelet/pods/e952a133-9bdb-4af5-8b16-38df266c53cd/volumes"
Nov 03 15:07:24.054440 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:24.054409    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"]
Nov 03 15:07:24.068373 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:24.068346    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tfvlc"]
Nov 03 15:07:26.066308 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:26.066276    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d92703d-4625-4afc-8794-810aa5a5abbe" path="/var/lib/kubelet/pods/1d92703d-4625-4afc-8794-810aa5a5abbe/volumes"
Nov 03 15:07:27.985393 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:07:27.985357    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:07:48.596070 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:48.595991    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:07:53.990410 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:53.990371    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"]
Nov 03 15:07:53.991005 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:53.990989    6481 cpu_manager.go:414] "containerMap: removing container" podUID="0274ec84-600c-4062-be83-dc32ddd769e0" containerName="revoke-keys"
Nov 03 15:07:53.991089 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:53.991011    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0274ec84-600c-4062-be83-dc32ddd769e0" containerName="revoke-keys"
Nov 03 15:07:53.991182 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:53.991166    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="0274ec84-600c-4062-be83-dc32ddd769e0" containerName="revoke-keys"
Nov 03 15:07:53.994360 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:53.994338    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.015399 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.015364    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"]
Nov 03 15:07:54.023551 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.023519    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:07:54.023551 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.023554    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:07:54.023720 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.023589    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:07:54.023720 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.023592    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:07:54.112656 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.112626    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.112822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.112684    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.112871 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.112841    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29gs4\" (UniqueName: \"kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.213545 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.213518    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29gs4\" (UniqueName: \"kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.213731 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.213576    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.213731 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.213633    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.214318 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.214292    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.215966 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.215944    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.235192 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.235166    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29gs4\" (UniqueName: \"kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4\") pod \"maas-api-revoke-sub-d576e68b-bw87d\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.308064 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.308009    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:54.609038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.609009    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"]
Nov 03 15:07:54.611353 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:07:54.611326    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bf9293_e677_49a7_b6a3_62ce2433b997.slice/crio-f0ea8b0b4c2e9c939c7998e92ed03b541464c5bb26b7199ad50212625b4be6ce WatchSource:0}: Error finding container f0ea8b0b4c2e9c939c7998e92ed03b541464c5bb26b7199ad50212625b4be6ce: Status 404 returned error can't find the container with id f0ea8b0b4c2e9c939c7998e92ed03b541464c5bb26b7199ad50212625b4be6ce
Nov 03 15:07:54.621930 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:54.621892    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d" event={"ID":"a7bf9293-e677-49a7-b6a3-62ce2433b997","Type":"ContainerStarted","Data":"f0ea8b0b4c2e9c939c7998e92ed03b541464c5bb26b7199ad50212625b4be6ce"}
Nov 03 15:07:55.628407 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:55.628371    6481 generic.go:356] "Generic (PLEG): container finished" podID="a7bf9293-e677-49a7-b6a3-62ce2433b997" containerID="6f5552b9bac18087cf2f210a300da18870d513422cd449af386dca9f6d350191" exitCode=0
Nov 03 15:07:55.628830 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:55.628419    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d" event={"ID":"a7bf9293-e677-49a7-b6a3-62ce2433b997","Type":"ContainerDied","Data":"6f5552b9bac18087cf2f210a300da18870d513422cd449af386dca9f6d350191"}
Nov 03 15:07:56.771296 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.771274    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:56.837652 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.837625    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca\") pod \"a7bf9293-e677-49a7-b6a3-62ce2433b997\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") "
Nov 03 15:07:56.837843 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.837687    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4\" (UniqueName: \"kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4\") pod \"a7bf9293-e677-49a7-b6a3-62ce2433b997\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") "
Nov 03 15:07:56.837843 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.837812    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca\") pod \"a7bf9293-e677-49a7-b6a3-62ce2433b997\" (UID: \"a7bf9293-e677-49a7-b6a3-62ce2433b997\") "
Nov 03 15:07:56.838033 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.838003    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca" pod "a7bf9293-e677-49a7-b6a3-62ce2433b997" (UID: "a7bf9293-e677-49a7-b6a3-62ce2433b997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:07:56.838256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.838219    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:07:56.839910 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.839883    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4" pod "a7bf9293-e677-49a7-b6a3-62ce2433b997" (UID: "a7bf9293-e677-49a7-b6a3-62ce2433b997"). InnerVolumeSpecName "kube-api-access-29gs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:07:56.840013 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.839908    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca" pod "a7bf9293-e677-49a7-b6a3-62ce2433b997" (UID: "a7bf9293-e677-49a7-b6a3-62ce2433b997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:07:56.939759 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.939734    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29gs4\" (UniqueName: \"kubernetes.io/projected/a7bf9293-e677-49a7-b6a3-62ce2433b997-kube-api-access-29gs4\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:07:56.939759 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:56.939761    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7bf9293-e677-49a7-b6a3-62ce2433b997-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:07:57.644632 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:57.644570    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"
Nov 03 15:07:57.644632 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:57.644611    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d" event={"ID":"a7bf9293-e677-49a7-b6a3-62ce2433b997","Type":"ContainerDied","Data":"f0ea8b0b4c2e9c939c7998e92ed03b541464c5bb26b7199ad50212625b4be6ce"}
Nov 03 15:07:57.644844 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:07:57.644707    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0ea8b0b4c2e9c939c7998e92ed03b541464c5bb26b7199ad50212625b4be6ce"
Nov 03 15:07:57.985269 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:07:57.985214    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:08:03.078055 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:03.078022    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:05.378151 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:05.378116    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:08:09.034968 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:09.034927    6481 scope.go:122] "RemoveContainer" containerID="3229f685ca7ebda8f10ac4dcddf26b853251a690c1318c7d3b02a17506f5f21b"
Nov 03 15:08:09.058542 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:09.058506    6481 scope.go:122] "RemoveContainer" containerID="65a26c333265e038d3202d835522d04e011232a3fc56f06108aa59cc5f3c5a6b"
Nov 03 15:08:09.071443 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:09.071410    6481 scope.go:122] "RemoveContainer" containerID="ef6b8ee95df0c6d885431ece1673b6dbf2dae1fc1f623b805fee6ee80b73eadb"
Nov 03 15:08:10.846253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:10.842678    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:08:13.156522 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.156494    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"]
Nov 03 15:08:13.156944 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.156928    6481 cpu_manager.go:414] "containerMap: removing container" podUID="a7bf9293-e677-49a7-b6a3-62ce2433b997" containerName="revoke-keys"
Nov 03 15:08:13.156996 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.156946    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7bf9293-e677-49a7-b6a3-62ce2433b997" containerName="revoke-keys"
Nov 03 15:08:13.157058 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.157047    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7bf9293-e677-49a7-b6a3-62ce2433b997" containerName="revoke-keys"
Nov 03 15:08:13.161911 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.161895    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.182641 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.182616    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"]
Nov 03 15:08:13.191665 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.191637    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:08:13.191665 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.191642    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:08:13.191861 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.191675    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:08:13.191861 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.191755    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:08:13.245379 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.245345    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.245540 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.245487    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpq2w\" (UniqueName: \"kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.245611 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.245541    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.346028 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.345999    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpq2w\" (UniqueName: \"kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.346168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.346039    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.346168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.346081    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.346827 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.346807    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.348397 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.348378    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.367312 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.367284    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpq2w\" (UniqueName: \"kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w\") pod \"maas-api-revoke-sub-d2bc93bc-dccvt\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.475169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.475143    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:13.775129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:13.775102    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"]
Nov 03 15:08:13.776604 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:08:13.776580    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2eb5f9_6544_4133_88e2_29db7766959a.slice/crio-3e6e8d977c765a8817321f37e081602011f5f28926187398213e748145adbf0b WatchSource:0}: Error finding container 3e6e8d977c765a8817321f37e081602011f5f28926187398213e748145adbf0b: Status 404 returned error can't find the container with id 3e6e8d977c765a8817321f37e081602011f5f28926187398213e748145adbf0b
Nov 03 15:08:14.786453 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:14.786371    6481 generic.go:356] "Generic (PLEG): container finished" podID="5d2eb5f9-6544-4133-88e2-29db7766959a" containerID="39aea15704546d72be0a3b1630c8e724530d15edb1525fabc32c5a8a71b133a2" exitCode=0
Nov 03 15:08:14.786453 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:14.786437    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt" event={"ID":"5d2eb5f9-6544-4133-88e2-29db7766959a","Type":"ContainerDied","Data":"39aea15704546d72be0a3b1630c8e724530d15edb1525fabc32c5a8a71b133a2"}
Nov 03 15:08:14.786839 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:14.786517    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt" event={"ID":"5d2eb5f9-6544-4133-88e2-29db7766959a","Type":"ContainerStarted","Data":"3e6e8d977c765a8817321f37e081602011f5f28926187398213e748145adbf0b"}
Nov 03 15:08:15.765789 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.765754    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:08:15.926905 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.926882    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:15.968129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.968099    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w\" (UniqueName: \"kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w\") pod \"5d2eb5f9-6544-4133-88e2-29db7766959a\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") "
Nov 03 15:08:15.968294 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.968217    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca\") pod \"5d2eb5f9-6544-4133-88e2-29db7766959a\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") "
Nov 03 15:08:15.968294 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.968282    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca\") pod \"5d2eb5f9-6544-4133-88e2-29db7766959a\" (UID: \"5d2eb5f9-6544-4133-88e2-29db7766959a\") "
Nov 03 15:08:15.968977 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.968952    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca" pod "5d2eb5f9-6544-4133-88e2-29db7766959a" (UID: "5d2eb5f9-6544-4133-88e2-29db7766959a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:08:15.970860 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.970833    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca" pod "5d2eb5f9-6544-4133-88e2-29db7766959a" (UID: "5d2eb5f9-6544-4133-88e2-29db7766959a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:08:15.970961 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:15.970932    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w" pod "5d2eb5f9-6544-4133-88e2-29db7766959a" (UID: "5d2eb5f9-6544-4133-88e2-29db7766959a"). InnerVolumeSpecName "kube-api-access-hpq2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:08:16.069492 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:16.069420    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:08:16.069492 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:16.069448    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpq2w\" (UniqueName: \"kubernetes.io/projected/5d2eb5f9-6544-4133-88e2-29db7766959a-kube-api-access-hpq2w\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:08:16.069492 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:16.069458    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d2eb5f9-6544-4133-88e2-29db7766959a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:08:16.797586 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:16.797555    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"
Nov 03 15:08:16.797771 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:16.797555    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt" event={"ID":"5d2eb5f9-6544-4133-88e2-29db7766959a","Type":"ContainerDied","Data":"3e6e8d977c765a8817321f37e081602011f5f28926187398213e748145adbf0b"}
Nov 03 15:08:16.797771 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:16.797677    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6e8d977c765a8817321f37e081602011f5f28926187398213e748145adbf0b"
Nov 03 15:08:22.038898 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:22.038864    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"]
Nov 03 15:08:22.050418 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:22.050392    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-vpgbb"]
Nov 03 15:08:22.066315 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:22.066284    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23e89c6-7340-442f-9e94-945e784275f4" path="/var/lib/kubelet/pods/f23e89c6-7340-442f-9e94-945e784275f4/volumes"
Nov 03 15:08:27.986028 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:08:27.985989    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:08:29.079192 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:29.079159    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:29.299204 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:29.299174    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:32.010418 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:32.010379    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:32.239302 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:32.239273    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:33.061927 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.061892    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"]
Nov 03 15:08:33.072601 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.072576    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xqq9p"]
Nov 03 15:08:33.751938 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.751902    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"]
Nov 03 15:08:33.752385 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.752371    6481 cpu_manager.go:414] "containerMap: removing container" podUID="5d2eb5f9-6544-4133-88e2-29db7766959a" containerName="revoke-keys"
Nov 03 15:08:33.752438 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.752388    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2eb5f9-6544-4133-88e2-29db7766959a" containerName="revoke-keys"
Nov 03 15:08:33.752484 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.752475    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2eb5f9-6544-4133-88e2-29db7766959a" containerName="revoke-keys"
Nov 03 15:08:33.756809 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.756789    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.775575 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.775554    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:08:33.775701 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.775658    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:08:33.787118 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.787096    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:08:33.787451 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.787429    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:08:33.789599 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.789566    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"]
Nov 03 15:08:33.845622 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.845585    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.845786 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.845717    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.845872 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.845788    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxdl2\" (UniqueName: \"kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.946367 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.946332    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.946538 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.946377    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxdl2\" (UniqueName: \"kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.946538 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.946466    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.947132 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.947106    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.948722 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.948698    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:33.968655 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:33.968634    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxdl2\" (UniqueName: \"kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2\") pod \"maas-api-revoke-sub-3e6552fd-mhptk\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:34.066072 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.066002    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bb95c23-d306-4019-b71c-514a13971cb2" path="/var/lib/kubelet/pods/6bb95c23-d306-4019-b71c-514a13971cb2/volumes"
Nov 03 15:08:34.070012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.069994    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:34.369865 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.369834    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"]
Nov 03 15:08:34.371482 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:08:34.371458    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5274376_256c_41d4_8041_4fdb300f2bab.slice/crio-3cf723443c0f644e413dda5fc0831ce9d29aba9143fbfabc315206adbd1c6381 WatchSource:0}: Error finding container 3cf723443c0f644e413dda5fc0831ce9d29aba9143fbfabc315206adbd1c6381: Status 404 returned error can't find the container with id 3cf723443c0f644e413dda5fc0831ce9d29aba9143fbfabc315206adbd1c6381
Nov 03 15:08:34.392781 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.392759    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:08:34.905077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.904987    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk" event={"ID":"d5274376-256c-41d4-8041-4fdb300f2bab","Type":"ContainerStarted","Data":"51b86a540ade7afa00196c6603ce88a5d567ac095e54392558b78b8a936dfeec"}
Nov 03 15:08:34.905077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.905052    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk" event={"ID":"d5274376-256c-41d4-8041-4fdb300f2bab","Type":"ContainerStarted","Data":"3cf723443c0f644e413dda5fc0831ce9d29aba9143fbfabc315206adbd1c6381"}
Nov 03 15:08:34.957533 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:34.957467    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk" podStartSLOduration=1.957447701 podStartE2EDuration="1.957447701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:08:34.953513807 +0000 UTC m=+3393.003544991" watchObservedRunningTime="2026-11-03 15:08:34.957447701 +0000 UTC m=+3393.007478887"
Nov 03 15:08:35.227606 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:35.227574    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:35.434309 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:35.434273    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:35.915487 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:35.915444    6481 generic.go:356] "Generic (PLEG): container finished" podID="d5274376-256c-41d4-8041-4fdb300f2bab" containerID="51b86a540ade7afa00196c6603ce88a5d567ac095e54392558b78b8a936dfeec" exitCode=0
Nov 03 15:08:35.915700 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:35.915527    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk" event={"ID":"d5274376-256c-41d4-8041-4fdb300f2bab","Type":"ContainerDied","Data":"51b86a540ade7afa00196c6603ce88a5d567ac095e54392558b78b8a936dfeec"}
Nov 03 15:08:37.615431 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.615400    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:37.713253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.710656    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca\") pod \"d5274376-256c-41d4-8041-4fdb300f2bab\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") "
Nov 03 15:08:37.720251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.714339    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca\") pod \"d5274376-256c-41d4-8041-4fdb300f2bab\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") "
Nov 03 15:08:37.720251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.714461    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2\" (UniqueName: \"kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2\") pod \"d5274376-256c-41d4-8041-4fdb300f2bab\" (UID: \"d5274376-256c-41d4-8041-4fdb300f2bab\") "
Nov 03 15:08:37.753839 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.753809    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca" pod "d5274376-256c-41d4-8041-4fdb300f2bab" (UID: "d5274376-256c-41d4-8041-4fdb300f2bab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:08:37.788532 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.788444    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2" pod "d5274376-256c-41d4-8041-4fdb300f2bab" (UID: "d5274376-256c-41d4-8041-4fdb300f2bab"). InnerVolumeSpecName "kube-api-access-mxdl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:08:37.851264 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.843950    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:08:37.851525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.851501    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxdl2\" (UniqueName: \"kubernetes.io/projected/d5274376-256c-41d4-8041-4fdb300f2bab-kube-api-access-mxdl2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:08:37.897777 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.897723    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca" pod "d5274376-256c-41d4-8041-4fdb300f2bab" (UID: "d5274376-256c-41d4-8041-4fdb300f2bab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:08:37.966099 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:37.966061    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5274376-256c-41d4-8041-4fdb300f2bab-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:08:38.023382 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:38.023344    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk" event={"ID":"d5274376-256c-41d4-8041-4fdb300f2bab","Type":"ContainerDied","Data":"3cf723443c0f644e413dda5fc0831ce9d29aba9143fbfabc315206adbd1c6381"}
Nov 03 15:08:38.023669 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:38.023648    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf723443c0f644e413dda5fc0831ce9d29aba9143fbfabc315206adbd1c6381"
Nov 03 15:08:38.023860 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:38.023844    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"
Nov 03 15:08:40.412010 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:40.411971    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:08:44.192824 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:44.192792    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:51.444275 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:08:51.444214    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:08:57.986090 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:08:57.986052    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:09:03.179157 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:03.179124    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:09:09.111432 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:09.111399    6481 scope.go:122] "RemoveContainer" containerID="cb93ff0e14eea3bbf53460d3b1aee3d4c0b75cdc0a2992a30cd48ff28fd277d8"
Nov 03 15:09:09.119999 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:09.119978    6481 scope.go:122] "RemoveContainer" containerID="c7a91a4f74b8b93e2530b069a1065517c44beed03b59656ba5d798595d0c7a04"
Nov 03 15:09:09.127931 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:09.127904    6481 scope.go:122] "RemoveContainer" containerID="106aa83dea016071ee5e1add5d115801612156152bf30487478d2ba0761d7df9"
Nov 03 15:09:27.986198 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:09:27.986115    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:09:50.656029 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:50.655997    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:09:57.056539 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:57.056505    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"]
Nov 03 15:09:57.068105 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:57.068076    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-038cdddc-scd2w"]
Nov 03 15:09:57.985575 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:09:57.985535    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:09:58.065389 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:09:58.065355    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d29685cb-fac6-4196-bfec-a3c672530a4b" path="/var/lib/kubelet/pods/d29685cb-fac6-4196-bfec-a3c672530a4b/volumes"
Nov 03 15:10:01.228668 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:01.228629    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:03.316435 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:03.316404    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:09.159115 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:09.159082    6481 scope.go:122] "RemoveContainer" containerID="6869c7e9e2b0f2cd83bd5f41870a10de49a1700c72d5c6917f4293a0cb34a733"
Nov 03 15:10:09.167985 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:09.167952    6481 scope.go:122] "RemoveContainer" containerID="b8d88ab2bf3cedb2844e3d8464020015f18b71c7f2c20ce340c619feed2663c3"
Nov 03 15:10:09.175685 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:09.175667    6481 scope.go:122] "RemoveContainer" containerID="aa26ade8f39ce3acb709b0b962ad5a5e24b704e09720e7e7d4c8173d817ec814"
Nov 03 15:10:14.705101 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:14.705050    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:14.941759 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:14.941732    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:15.675978 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:15.675933    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:27.985877 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:10:27.985831    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:10:41.876976 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:41.876947    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:49.179171 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:10:49.179136    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:10:57.986310 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:10:57.986270    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:11:03.449534 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:11:03.449504    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:11:09.203295 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:11:09.203264    6481 scope.go:122] "RemoveContainer" containerID="6308d8c2088872d042dfb1ca102c9d2a7291be806698d921b25cc1b0cc98acf7"
Nov 03 15:11:14.044804 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:11:14.044767    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"]
Nov 03 15:11:14.056361 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:11:14.056333    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-wjg86"]
Nov 03 15:11:14.065617 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:11:14.065587    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0274ec84-600c-4062-be83-dc32ddd769e0" path="/var/lib/kubelet/pods/0274ec84-600c-4062-be83-dc32ddd769e0/volumes"
Nov 03 15:11:27.984541 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:11:27.984499    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:11:57.985198 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:11:57.985151    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:12:03.558056 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:03.558021    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:12:09.224091 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:09.224058    6481 scope.go:122] "RemoveContainer" containerID="e9b7aa718f2eb52df40f804985b01d327da76c1199b69cc1cb2eaf9b6080413e"
Nov 03 15:12:12.383015 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:12.382983    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:12:12.604663 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:12.604626    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:12:27.985145 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:12:27.985106    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:12:28.845623 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:28.845592    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:12:35.814486 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:35.814454    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:12:44.055278 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:44.055247    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:12:57.045780 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:57.045737    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"]
Nov 03 15:12:57.064487 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:57.064448    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bw87d"]
Nov 03 15:12:57.986508 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:12:57.986473    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:12:58.066236 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:12:58.066194    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7bf9293-e677-49a7-b6a3-62ce2433b997" path="/var/lib/kubelet/pods/a7bf9293-e677-49a7-b6a3-62ce2433b997/volumes"
Nov 03 15:13:03.701784 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:03.701750    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:13:09.251073 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:09.251045    6481 scope.go:122] "RemoveContainer" containerID="ff0532cb11434095857e9fa09b62e2796ef5b7888713d48e967e8d75f98c27e1"
Nov 03 15:13:16.042763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:16.041382    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"]
Nov 03 15:13:16.052355 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:16.052325    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dccvt"]
Nov 03 15:13:16.065511 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:16.065483    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2eb5f9-6544-4133-88e2-29db7766959a" path="/var/lib/kubelet/pods/5d2eb5f9-6544-4133-88e2-29db7766959a/volumes"
Nov 03 15:13:27.985839 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:13:27.985802    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:13:37.052772 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:37.052730    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"]
Nov 03 15:13:37.072472 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:37.072445    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e6552fd-mhptk"]
Nov 03 15:13:38.065689 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:38.065656    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5274376-256c-41d4-8041-4fdb300f2bab" path="/var/lib/kubelet/pods/d5274376-256c-41d4-8041-4fdb300f2bab/volumes"
Nov 03 15:13:57.986368 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:13:57.986283    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:13:59.004029 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:13:59.003994    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:14:00.869081 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:00.869046    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:14:03.795028 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:03.794995    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:14:09.270997 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:09.270966    6481 scope.go:122] "RemoveContainer" containerID="6f5552b9bac18087cf2f210a300da18870d513422cd449af386dca9f6d350191"
Nov 03 15:14:09.861015 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:09.860986    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:14:10.719127 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:10.719090    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:14:13.443472 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.443437    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"]
Nov 03 15:14:13.443998 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.443660    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" containerID="cri-o://74ee0abadbdf9d9992906b7f247e106ef9dce51829434178950dc52dd4bb26e6" gracePeriod=30
Nov 03 15:14:13.454928 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.454904    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"]
Nov 03 15:14:13.455116 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.455082    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" containerID="cri-o://98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273" gracePeriod=30
Nov 03 15:14:13.457330 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.457307    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"]
Nov 03 15:14:13.457531 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.457496    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" containerID="cri-o://c6ebd796b15d03c20c182f171f87ae4a9ceb8091fa37ade57dfafd7538ed2e5b" gracePeriod=30
Nov 03 15:14:13.466059 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.466038    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"]
Nov 03 15:14:13.466200 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:13.466177    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" containerID="cri-o://dd6d903bce174570abe23a5de41e873ba1c9664c64d0a65d2c2e68d130c74a63" gracePeriod=30
Nov 03 15:14:17.119800 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:17.119762    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z57r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:14:17.120191 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:17.119821    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:14:17.176115 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:17.176084    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:14:17.176292 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:17.176148    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:14:17.243090 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:17.243053    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-r88l9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:14:17.243212 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:17.243101    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:14:18.132741 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.132704    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:14:18.133139 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.132770    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:14:18.142576 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.142538    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:14:18.742675 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.742651    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 15:14:18.756975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.756943    6481 generic.go:356] "Generic (PLEG): container finished" podID="c792c6d3-b155-4230-8458-964c5944780d" containerID="98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273" exitCode=0
Nov 03 15:14:18.757151 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.757000    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" event={"ID":"c792c6d3-b155-4230-8458-964c5944780d","Type":"ContainerDied","Data":"98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273"}
Nov 03 15:14:18.757151 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.757065    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh" event={"ID":"c792c6d3-b155-4230-8458-964c5944780d","Type":"ContainerDied","Data":"9e3f499bb1880a068a6e996feec4160b0fffd1925c6a38474ade716b8a781b87"}
Nov 03 15:14:18.757151 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.757097    6481 scope.go:122] "RemoveContainer" containerID="98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273"
Nov 03 15:14:18.757372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.757010    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"
Nov 03 15:14:18.760102 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.759891    6481 generic.go:356] "Generic (PLEG): container finished" podID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerID="dd6d903bce174570abe23a5de41e873ba1c9664c64d0a65d2c2e68d130c74a63" exitCode=0
Nov 03 15:14:18.760102 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.760030    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" event={"ID":"7426741c-5396-40d8-acbb-90dc68aa5bdd","Type":"ContainerDied","Data":"dd6d903bce174570abe23a5de41e873ba1c9664c64d0a65d2c2e68d130c74a63"}
Nov 03 15:14:18.761874 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.761848    6481 generic.go:356] "Generic (PLEG): container finished" podID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerID="74ee0abadbdf9d9992906b7f247e106ef9dce51829434178950dc52dd4bb26e6" exitCode=0
Nov 03 15:14:18.761976 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.761916    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" event={"ID":"f26794db-ebe2-44b0-aca5-5b2bf5374762","Type":"ContainerDied","Data":"74ee0abadbdf9d9992906b7f247e106ef9dce51829434178950dc52dd4bb26e6"}
Nov 03 15:14:18.763312 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.763294    6481 generic.go:356] "Generic (PLEG): container finished" podID="df8240a2-be3d-4408-880d-0512f957be94" containerID="c6ebd796b15d03c20c182f171f87ae4a9ceb8091fa37ade57dfafd7538ed2e5b" exitCode=0
Nov 03 15:14:18.763433 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.763329    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" event={"ID":"df8240a2-be3d-4408-880d-0512f957be94","Type":"ContainerDied","Data":"c6ebd796b15d03c20c182f171f87ae4a9ceb8091fa37ade57dfafd7538ed2e5b"}
Nov 03 15:14:18.763957 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.763930    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764057 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.763985    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764057 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764019    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764185 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764060    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764185 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764107    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764185 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764155    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764454 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764252    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764454 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764292    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764454 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764340    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs\") pod \"c792c6d3-b155-4230-8458-964c5944780d\" (UID: \"c792c6d3-b155-4230-8458-964c5944780d\") "
Nov 03 15:14:18.764620 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764496    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.765055 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764772    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:14:18.765055 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764844    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.765055 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.764938    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.765383 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.765314    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.765383 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.765341    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.767397 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.767372    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:14:18.769650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.769629    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "kube-api-access-zh5c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.769750 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.769673    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.769750 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.769696    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy" pod "c792c6d3-b155-4230-8458-964c5944780d" (UID: "c792c6d3-b155-4230-8458-964c5944780d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.772901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.772878    6481 scope.go:122] "RemoveContainer" containerID="98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273"
Nov 03 15:14:18.773179 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:14:18.773157    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273\": container with ID starting with 98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273 not found: ID does not exist" containerID="98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273"
Nov 03 15:14:18.773387 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.773189    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273"} err="failed to get container status \"98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273\": rpc error: code = NotFound desc = could not find container \"98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273\": container with ID starting with 98cde6aa0326a7d5f6d48b53da6ddfdd1778bfacb556ce49cf02e4d79686b273 not found: ID does not exist"
Nov 03 15:14:18.776393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.776374    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 15:14:18.780890 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.780852    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 15:14:18.784160 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.784144    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 15:14:18.865280 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865252    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.865460 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865288    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.865460 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865331    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.865591 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865502    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.865591 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865559    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.865591 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865577    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.865763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865615    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.865763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865647    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.865763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865676    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.865763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865711    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.865763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865745    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.866038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865793    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.866038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865832    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.866038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865884    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.866038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865920    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.866038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.865952    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.866038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866007    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866050    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866082    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866114    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866145    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866183    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866216    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866250    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866291    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866325    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data\") pod \"df8240a2-be3d-4408-880d-0512f957be94\" (UID: \"df8240a2-be3d-4408-880d-0512f957be94\") "
Nov 03 15:14:18.866393 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866372    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo\") pod \"7426741c-5396-40d8-acbb-90dc68aa5bdd\" (UID: \"7426741c-5396-40d8-acbb-90dc68aa5bdd\") "
Nov 03 15:14:18.866934 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866432    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.866934 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866461    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert\") pod \"f26794db-ebe2-44b0-aca5-5b2bf5374762\" (UID: \"f26794db-ebe2-44b0-aca5-5b2bf5374762\") "
Nov 03 15:14:18.866934 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866522    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.866934 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866736    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866938    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866961    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df8240a2-be3d-4408-880d-0512f957be94-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866976    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh5c5\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-kube-api-access-zh5c5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866984    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.866992    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867040    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c792c6d3-b155-4230-8458-964c5944780d-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867057    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867071    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867086    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867099    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c792c6d3-b155-4230-8458-964c5944780d-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867112    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c792c6d3-b155-4230-8458-964c5944780d-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867125    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c792c6d3-b155-4230-8458-964c5944780d-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867138    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.867178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867065    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867377    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867413    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.867692    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.868114    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.868136    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.868390    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.868395    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.868470    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.868627    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.870620 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.870598    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:14:18.870827 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.870804    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:14:18.871212 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.871190    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "kube-api-access-lc4xd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.871353 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.871306    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.871575 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.871554    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:14:18.872446 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.872426    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "kube-api-access-ngpxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.872562 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.872546    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.872619 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.872557    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy" pod "f26794db-ebe2-44b0-aca5-5b2bf5374762" (UID: "f26794db-ebe2-44b0-aca5-5b2bf5374762"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.872619 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.872568    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.872619 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.872577    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "kube-api-access-7b5h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:14:18.872619 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.872582    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy" pod "df8240a2-be3d-4408-880d-0512f957be94" (UID: "df8240a2-be3d-4408-880d-0512f957be94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.873784 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.873770    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy" pod "7426741c-5396-40d8-acbb-90dc68aa5bdd" (UID: "7426741c-5396-40d8-acbb-90dc68aa5bdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:14:18.968626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968592    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968619    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968632    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968644    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968657    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968668    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968678    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968690    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968702    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df8240a2-be3d-4408-880d-0512f957be94-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968714    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngpxs\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-kube-api-access-ngpxs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968729    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7426741c-5396-40d8-acbb-90dc68aa5bdd-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968741    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968753    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968766    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f26794db-ebe2-44b0-aca5-5b2bf5374762-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968777    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7426741c-5396-40d8-acbb-90dc68aa5bdd-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968788    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968799    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7426741c-5396-40d8-acbb-90dc68aa5bdd-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968811    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f26794db-ebe2-44b0-aca5-5b2bf5374762-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968823    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f26794db-ebe2-44b0-aca5-5b2bf5374762-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968836    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968849    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b5h5\" (UniqueName: \"kubernetes.io/projected/7426741c-5396-40d8-acbb-90dc68aa5bdd-kube-api-access-7b5h5\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968861    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df8240a2-be3d-4408-880d-0512f957be94-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:18.968922 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:18.968875    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc4xd\" (UniqueName: \"kubernetes.io/projected/df8240a2-be3d-4408-880d-0512f957be94-kube-api-access-lc4xd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:14:19.106199 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.106168    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"]
Nov 03 15:14:19.113038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.113014    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-qpbjh"]
Nov 03 15:14:19.769640 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.769565    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"
Nov 03 15:14:19.769640 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.769590    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9" event={"ID":"df8240a2-be3d-4408-880d-0512f957be94","Type":"ContainerDied","Data":"9a6048c9f14fb1466dd9d4fa9c1a07f5315ae807ac1bb90812dc02c380ac875c"}
Nov 03 15:14:19.770199 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.769651    6481 scope.go:122] "RemoveContainer" containerID="c6ebd796b15d03c20c182f171f87ae4a9ceb8091fa37ade57dfafd7538ed2e5b"
Nov 03 15:14:19.772187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.772168    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"
Nov 03 15:14:19.772319 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.772208    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2" event={"ID":"7426741c-5396-40d8-acbb-90dc68aa5bdd","Type":"ContainerDied","Data":"14d6c23855a0628a05f5233efd344ea46f80d4c6b7a4ff055a44fcb7dd4202d1"}
Nov 03 15:14:19.773655 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.773629    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq" event={"ID":"f26794db-ebe2-44b0-aca5-5b2bf5374762","Type":"ContainerDied","Data":"d8097ef083c06fbf858e04f4428c03d30cb3aa2a6344df57c4f434d72bda1d2e"}
Nov 03 15:14:19.773767 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.773654    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"
Nov 03 15:14:19.780102 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.779996    6481 scope.go:122] "RemoveContainer" containerID="dd6d903bce174570abe23a5de41e873ba1c9664c64d0a65d2c2e68d130c74a63"
Nov 03 15:14:19.789648 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.789633    6481 scope.go:122] "RemoveContainer" containerID="74ee0abadbdf9d9992906b7f247e106ef9dce51829434178950dc52dd4bb26e6"
Nov 03 15:14:19.824293 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.824269    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"]
Nov 03 15:14:19.835772 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.835749    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-mcrdq"]
Nov 03 15:14:19.868789 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.868765    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"]
Nov 03 15:14:19.879785 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.879766    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z57r2"]
Nov 03 15:14:19.916667 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.916641    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"]
Nov 03 15:14:19.925626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:19.925608    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-r88l9"]
Nov 03 15:14:20.065515 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:20.065443    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" path="/var/lib/kubelet/pods/7426741c-5396-40d8-acbb-90dc68aa5bdd/volumes"
Nov 03 15:14:20.065903 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:20.065890    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c792c6d3-b155-4230-8458-964c5944780d" path="/var/lib/kubelet/pods/c792c6d3-b155-4230-8458-964c5944780d/volumes"
Nov 03 15:14:20.066301 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:20.066289    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8240a2-be3d-4408-880d-0512f957be94" path="/var/lib/kubelet/pods/df8240a2-be3d-4408-880d-0512f957be94/volumes"
Nov 03 15:14:20.066983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:14:20.066969    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" path="/var/lib/kubelet/pods/f26794db-ebe2-44b0-aca5-5b2bf5374762/volumes"
Nov 03 15:14:27.985122 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:14:27.985080    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:14:57.985005 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:14:57.984968    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:15:00.158648 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.158617    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"]
Nov 03 15:15:00.159024 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.158976    6481 cpu_manager.go:414] "containerMap: removing container" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy"
Nov 03 15:15:00.159024 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.158988    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy"
Nov 03 15:15:00.159024 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159004    6481 cpu_manager.go:414] "containerMap: removing container" podUID="d5274376-256c-41d4-8041-4fdb300f2bab" containerName="revoke-keys"
Nov 03 15:15:00.159024 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159009    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5274376-256c-41d4-8041-4fdb300f2bab" containerName="revoke-keys"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159027    6481 cpu_manager.go:414] "containerMap: removing container" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159033    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159044    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159049    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159058    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159063    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159124    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="7426741c-5396-40d8-acbb-90dc68aa5bdd" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159132    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c792c6d3-b155-4230-8458-964c5944780d" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159139    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f26794db-ebe2-44b0-aca5-5b2bf5374762" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159147    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8240a2-be3d-4408-880d-0512f957be94" containerName="istio-proxy"
Nov 03 15:15:00.159169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.159153    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5274376-256c-41d4-8041-4fdb300f2bab" containerName="revoke-keys"
Nov 03 15:15:00.162285 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.162269    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.181417 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.181389    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:15:00.181679 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.181661    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:15:00.181793 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.181739    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:15:00.182306 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.182284    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"]
Nov 03 15:15:00.249438 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.249407    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895315-9fsdl\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.249585 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.249465    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42cfb\" (UniqueName: \"kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb\") pod \"maas-api-key-cleanup-29895315-9fsdl\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.350409 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.350369    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42cfb\" (UniqueName: \"kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb\") pod \"maas-api-key-cleanup-29895315-9fsdl\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.350565 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.350465    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895315-9fsdl\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.351043 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.351023    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895315-9fsdl\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.371354 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.371325    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42cfb\" (UniqueName: \"kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb\") pod \"maas-api-key-cleanup-29895315-9fsdl\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.474918 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.474835    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:00.772108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.772083    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"]
Nov 03 15:15:00.773933 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:15:00.773907    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf764d5_e2ad_4827_9908_06a02c1e0ad5.slice/crio-6a4307342bc99838779df55f665b8324a1a41877911b048f9b6706786095ac9d WatchSource:0}: Error finding container 6a4307342bc99838779df55f665b8324a1a41877911b048f9b6706786095ac9d: Status 404 returned error can't find the container with id 6a4307342bc99838779df55f665b8324a1a41877911b048f9b6706786095ac9d
Nov 03 15:15:00.978119 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:00.978083    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" event={"ID":"fdf764d5-e2ad-4827-9908-06a02c1e0ad5","Type":"ContainerStarted","Data":"6a4307342bc99838779df55f665b8324a1a41877911b048f9b6706786095ac9d"}
Nov 03 15:15:01.984417 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:01.984389    6481 generic.go:356] "Generic (PLEG): container finished" podID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerID="57d02f987a09ba78f6d847da14cfea4d368d5713b7dcfc15e2fd78555d8e4923" exitCode=60
Nov 03 15:15:01.984853 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:01.984463    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" event={"ID":"fdf764d5-e2ad-4827-9908-06a02c1e0ad5","Type":"ContainerDied","Data":"57d02f987a09ba78f6d847da14cfea4d368d5713b7dcfc15e2fd78555d8e4923"}
Nov 03 15:15:01.984853 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:01.984650    6481 scope.go:122] "RemoveContainer" containerID="57d02f987a09ba78f6d847da14cfea4d368d5713b7dcfc15e2fd78555d8e4923"
Nov 03 15:15:02.991582 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:02.991538    6481 generic.go:356] "Generic (PLEG): container finished" podID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerID="ccd78206f3a968d4d991576c93137779d0293082b6466e1ec505abe24bad4c28" exitCode=60
Nov 03 15:15:02.991975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:02.991599    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" event={"ID":"fdf764d5-e2ad-4827-9908-06a02c1e0ad5","Type":"ContainerDied","Data":"ccd78206f3a968d4d991576c93137779d0293082b6466e1ec505abe24bad4c28"}
Nov 03 15:15:02.991975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:02.991684    6481 scope.go:122] "RemoveContainer" containerID="57d02f987a09ba78f6d847da14cfea4d368d5713b7dcfc15e2fd78555d8e4923"
Nov 03 15:15:02.991975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:02.991890    6481 scope.go:122] "RemoveContainer" containerID="ccd78206f3a968d4d991576c93137779d0293082b6466e1ec505abe24bad4c28"
Nov 03 15:15:02.992156 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:15:02.992136    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895315-9fsdl_odh-ai-gateway-infra(fdf764d5-e2ad-4827-9908-06a02c1e0ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5"
Nov 03 15:15:03.985775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:03.985728    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:03.999014 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:03.998991    6481 scope.go:122] "RemoveContainer" containerID="ccd78206f3a968d4d991576c93137779d0293082b6466e1ec505abe24bad4c28"
Nov 03 15:15:03.999435 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:15:03.999245    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895315-9fsdl_odh-ai-gateway-infra(fdf764d5-e2ad-4827-9908-06a02c1e0ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5"
Nov 03 15:15:09.291213 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:09.291174    6481 scope.go:122] "RemoveContainer" containerID="39aea15704546d72be0a3b1630c8e724530d15edb1525fabc32c5a8a71b133a2"
Nov 03 15:15:09.299274 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:09.299255    6481 scope.go:122] "RemoveContainer" containerID="51b86a540ade7afa00196c6603ce88a5d567ac095e54392558b78b8a936dfeec"
Nov 03 15:15:18.060609 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:18.060526    6481 scope.go:122] "RemoveContainer" containerID="ccd78206f3a968d4d991576c93137779d0293082b6466e1ec505abe24bad4c28"
Nov 03 15:15:19.072060 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:19.071977    6481 generic.go:356] "Generic (PLEG): container finished" podID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerID="1d87fb297e417dcae7868069bd8c241ae18cb274b3a54276076032908b0f9837" exitCode=60
Nov 03 15:15:19.072060 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:19.072048    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" event={"ID":"fdf764d5-e2ad-4827-9908-06a02c1e0ad5","Type":"ContainerDied","Data":"1d87fb297e417dcae7868069bd8c241ae18cb274b3a54276076032908b0f9837"}
Nov 03 15:15:19.072608 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:19.072104    6481 scope.go:122] "RemoveContainer" containerID="ccd78206f3a968d4d991576c93137779d0293082b6466e1ec505abe24bad4c28"
Nov 03 15:15:19.072608 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:19.072446    6481 scope.go:122] "RemoveContainer" containerID="1d87fb297e417dcae7868069bd8c241ae18cb274b3a54276076032908b0f9837"
Nov 03 15:15:19.072723 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:15:19.072701    6481 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895315-9fsdl_odh-ai-gateway-infra(fdf764d5-e2ad-4827-9908-06a02c1e0ad5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5"
Nov 03 15:15:20.126539 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.126510    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"]
Nov 03 15:15:20.259670 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.259641    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:20.340866 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.340839    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb\" (UniqueName: \"kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb\") pod \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") "
Nov 03 15:15:20.341048 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.340928    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca\") pod \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\" (UID: \"fdf764d5-e2ad-4827-9908-06a02c1e0ad5\") "
Nov 03 15:15:20.341331 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.341302    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca" pod "fdf764d5-e2ad-4827-9908-06a02c1e0ad5" (UID: "fdf764d5-e2ad-4827-9908-06a02c1e0ad5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:15:20.342901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.342879    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb" pod "fdf764d5-e2ad-4827-9908-06a02c1e0ad5" (UID: "fdf764d5-e2ad-4827-9908-06a02c1e0ad5"). InnerVolumeSpecName "kube-api-access-42cfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:15:20.441951 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.441920    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42cfb\" (UniqueName: \"kubernetes.io/projected/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-kube-api-access-42cfb\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:15:20.441951 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:20.441948    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdf764d5-e2ad-4827-9908-06a02c1e0ad5-maas-api-service-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:15:21.082906 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:21.082868    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl" event={"ID":"fdf764d5-e2ad-4827-9908-06a02c1e0ad5","Type":"ContainerDied","Data":"6a4307342bc99838779df55f665b8324a1a41877911b048f9b6706786095ac9d"}
Nov 03 15:15:21.082906 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:21.082889    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"
Nov 03 15:15:21.083189 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:21.082928    6481 scope.go:122] "RemoveContainer" containerID="1d87fb297e417dcae7868069bd8c241ae18cb274b3a54276076032908b0f9837"
Nov 03 15:15:21.134618 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:21.134588    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"]
Nov 03 15:15:21.140876 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:21.140852    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895315-9fsdl"]
Nov 03 15:15:22.065613 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:22.065583    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" path="/var/lib/kubelet/pods/fdf764d5-e2ad-4827-9908-06a02c1e0ad5/volumes"
Nov 03 15:15:25.364862 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:25.364830    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:27.984697 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:15:27.984661    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:15:41.203627 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:41.203590    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:41.432186 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:41.432155    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:42.525301 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:42.525260    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:42.759223 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:42.759184    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:43.406757 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:43.406722    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:15:44.251678 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.251650    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"]
Nov 03 15:15:44.252078 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.252022    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:15:44.252078 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.252032    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:15:44.252078 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.252044    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:15:44.252078 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.252049    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:15:44.252242 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.252125    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:15:44.252242 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.252133    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:15:44.255329 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.255306    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.270124 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.270101    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:15:44.270255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.270188    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:15:44.281975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.281952    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:15:44.282158 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.282137    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:15:44.283708 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.283689    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"]
Nov 03 15:15:44.360650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.360618    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.360776 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.360656    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxzx\" (UniqueName: \"kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.360776 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.360757    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.462103 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.462064    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.462359 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.462171    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.462359 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.462192    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxzx\" (UniqueName: \"kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.462850 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.462829    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.464612 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.464581    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.482731 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.482699    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxzx\" (UniqueName: \"kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx\") pod \"maas-api-revoke-sub-1ead6fc6-68mb2\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.568405 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.568338    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:44.865541 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:44.865518    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"]
Nov 03 15:15:44.867913 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:15:44.867887    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b9606c_241e_4443_b9d1_93e3143fa707.slice/crio-839e70495263d91e91daad4a0f26482d2631fa67dacfe946f06d1dce51c6ba27 WatchSource:0}: Error finding container 839e70495263d91e91daad4a0f26482d2631fa67dacfe946f06d1dce51c6ba27: Status 404 returned error can't find the container with id 839e70495263d91e91daad4a0f26482d2631fa67dacfe946f06d1dce51c6ba27
Nov 03 15:15:45.193459 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:45.193425    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2" event={"ID":"60b9606c-241e-4443-b9d1-93e3143fa707","Type":"ContainerStarted","Data":"839e70495263d91e91daad4a0f26482d2631fa67dacfe946f06d1dce51c6ba27"}
Nov 03 15:15:46.199557 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:46.199526    6481 generic.go:356] "Generic (PLEG): container finished" podID="60b9606c-241e-4443-b9d1-93e3143fa707" containerID="2c4274f4be73ee41da446a8f122a3699ccfc6e3d9b17f44d16dc4daed75aa75d" exitCode=0
Nov 03 15:15:46.199967 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:46.199616    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2" event={"ID":"60b9606c-241e-4443-b9d1-93e3143fa707","Type":"ContainerDied","Data":"2c4274f4be73ee41da446a8f122a3699ccfc6e3d9b17f44d16dc4daed75aa75d"}
Nov 03 15:15:47.343490 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.343467    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:47.389044 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.389006    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca\") pod \"60b9606c-241e-4443-b9d1-93e3143fa707\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") "
Nov 03 15:15:47.389256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.389057    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca\") pod \"60b9606c-241e-4443-b9d1-93e3143fa707\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") "
Nov 03 15:15:47.389420 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.389397    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx\" (UniqueName: \"kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx\") pod \"60b9606c-241e-4443-b9d1-93e3143fa707\" (UID: \"60b9606c-241e-4443-b9d1-93e3143fa707\") "
Nov 03 15:15:47.389625 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.389418    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca" pod "60b9606c-241e-4443-b9d1-93e3143fa707" (UID: "60b9606c-241e-4443-b9d1-93e3143fa707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:15:47.390129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.390078    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:15:47.391413 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.391384    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca" pod "60b9606c-241e-4443-b9d1-93e3143fa707" (UID: "60b9606c-241e-4443-b9d1-93e3143fa707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:15:47.391522 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.391500    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx" pod "60b9606c-241e-4443-b9d1-93e3143fa707" (UID: "60b9606c-241e-4443-b9d1-93e3143fa707"). InnerVolumeSpecName "kube-api-access-zqxzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:15:47.491204 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.491113    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60b9606c-241e-4443-b9d1-93e3143fa707-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:15:47.491204 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:47.491147    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxzx\" (UniqueName: \"kubernetes.io/projected/60b9606c-241e-4443-b9d1-93e3143fa707-kube-api-access-zqxzx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:15:48.210221 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:48.210180    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2" event={"ID":"60b9606c-241e-4443-b9d1-93e3143fa707","Type":"ContainerDied","Data":"839e70495263d91e91daad4a0f26482d2631fa67dacfe946f06d1dce51c6ba27"}
Nov 03 15:15:48.210398 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:48.210191    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"
Nov 03 15:15:48.210398 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:48.210253    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="839e70495263d91e91daad4a0f26482d2631fa67dacfe946f06d1dce51c6ba27"
Nov 03 15:15:54.743099 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:15:54.743045    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:15:57.984464 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:15:57.984425    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:16:03.022250 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:03.022204    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:16:04.025844 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:04.025811    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:16:13.812915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.812881    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"]
Nov 03 15:16:13.813394 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.813307    6481 cpu_manager.go:414] "containerMap: removing container" podUID="60b9606c-241e-4443-b9d1-93e3143fa707" containerName="revoke-keys"
Nov 03 15:16:13.813394 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.813320    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b9606c-241e-4443-b9d1-93e3143fa707" containerName="revoke-keys"
Nov 03 15:16:13.813394 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.813348    6481 cpu_manager.go:414] "containerMap: removing container" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:16:13.813394 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.813354    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:16:13.813547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.813416    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdf764d5-e2ad-4827-9908-06a02c1e0ad5" containerName="cleanup"
Nov 03 15:16:13.813547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.813428    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b9606c-241e-4443-b9d1-93e3143fa707" containerName="revoke-keys"
Nov 03 15:16:13.817755 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.817735    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.829900 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.829879    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"]
Nov 03 15:16:13.833193 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.833177    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.840322 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.840303    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4nspm\""
Nov 03 15:16:13.842785 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.842763    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"]
Nov 03 15:16:13.856734 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.856710    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"]
Nov 03 15:16:13.920308 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920278    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.920493 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920314    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920493 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920338    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.920493 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920403    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4cc\" (UniqueName: \"kubernetes.io/projected/7285fd08-91c8-4113-9f66-d5e6810b02d5-kube-api-access-bp4cc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920531    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920556    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.920643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920606    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.920643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920637    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920654    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.920835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920671    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920691    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920796    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.920835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920812    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvdh\" (UniqueName: \"kubernetes.io/projected/0b5ad832-aa94-4941-9b2e-852a9ddfda03-kube-api-access-ncvdh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.920835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920832    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7285fd08-91c8-4113-9f66-d5e6810b02d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:13.921153 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920853    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.921153 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920886    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.921153 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.920963    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:13.921153 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:13.921030    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.021751 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021718    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.021943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021763    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.021943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021793    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.021943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021834    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4cc\" (UniqueName: \"kubernetes.io/projected/7285fd08-91c8-4113-9f66-d5e6810b02d5-kube-api-access-bp4cc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.021943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021880    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.021943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021903    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.021943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021934    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021969    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.021996    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022021    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022055    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022113    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022138    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvdh\" (UniqueName: \"kubernetes.io/projected/0b5ad832-aa94-4941-9b2e-852a9ddfda03-kube-api-access-ncvdh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022174    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7285fd08-91c8-4113-9f66-d5e6810b02d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022214    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022262    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022279    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022341 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022291    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022343    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022409    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022554    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022644    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022662    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022717    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.022949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022812    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.023381 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.022960    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.023381 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.023055    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.023516 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.023460    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7285fd08-91c8-4113-9f66-d5e6810b02d5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.024807 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.024783    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.024924 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.024906    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.024968 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.024947    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.025006 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.024947    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.043616 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.043589    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4cc\" (UniqueName: \"kubernetes.io/projected/7285fd08-91c8-4113-9f66-d5e6810b02d5-kube-api-access-bp4cc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.043708 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.043640    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvdh\" (UniqueName: \"kubernetes.io/projected/0b5ad832-aa94-4941-9b2e-852a9ddfda03-kube-api-access-ncvdh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.044120 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.044095    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7285fd08-91c8-4113-9f66-d5e6810b02d5-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-z6fd6\" (UID: \"7285fd08-91c8-4113-9f66-d5e6810b02d5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.044207 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.044150    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5ad832-aa94-4941-9b2e-852a9ddfda03-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-2p5f4\" (UID: \"0b5ad832-aa94-4941-9b2e-852a9ddfda03\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.131114 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.131043    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:14.146530 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.146506    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:14.450222 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.450188    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"]
Nov 03 15:16:14.464882 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:16:14.464855    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7285fd08_91c8_4113_9f66_d5e6810b02d5.slice/crio-7f4a23516296801604fd89c4a6d1702e4b65735c2d69425e3b57442daaa6f488 WatchSource:0}: Error finding container 7f4a23516296801604fd89c4a6d1702e4b65735c2d69425e3b57442daaa6f488: Status 404 returned error can't find the container with id 7f4a23516296801604fd89c4a6d1702e4b65735c2d69425e3b57442daaa6f488
Nov 03 15:16:14.473439 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.470641    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:14.473439 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.470782    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:14.473439 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.470867    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:14.491155 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.491119    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"]
Nov 03 15:16:14.510354 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:16:14.510329    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5ad832_aa94_4941_9b2e_852a9ddfda03.slice/crio-c4e501f052cf5ae2aef76085fdb6006f85df6a9938806583bd298868cf6cada4 WatchSource:0}: Error finding container c4e501f052cf5ae2aef76085fdb6006f85df6a9938806583bd298868cf6cada4: Status 404 returned error can't find the container with id c4e501f052cf5ae2aef76085fdb6006f85df6a9938806583bd298868cf6cada4
Nov 03 15:16:14.512616 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.512585    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:14.512733 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.512661    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:14.512733 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:14.512707    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:15.350465 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:15.350426    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" event={"ID":"0b5ad832-aa94-4941-9b2e-852a9ddfda03","Type":"ContainerStarted","Data":"a2de91e8c6aa665874b366e6e4812c2925f998f9e1ba12fdc9a20f78d847148d"}
Nov 03 15:16:15.351081 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:15.351057    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" event={"ID":"0b5ad832-aa94-4941-9b2e-852a9ddfda03","Type":"ContainerStarted","Data":"c4e501f052cf5ae2aef76085fdb6006f85df6a9938806583bd298868cf6cada4"}
Nov 03 15:16:15.354247 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:15.353578    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" event={"ID":"7285fd08-91c8-4113-9f66-d5e6810b02d5","Type":"ContainerStarted","Data":"4ee0c746dbd35f2111cb732e1b0c78306ca87b8a256f1c447bc4530812cc6c89"}
Nov 03 15:16:15.354247 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:15.353616    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" event={"ID":"7285fd08-91c8-4113-9f66-d5e6810b02d5","Type":"ContainerStarted","Data":"7f4a23516296801604fd89c4a6d1702e4b65735c2d69425e3b57442daaa6f488"}
Nov 03 15:16:15.419559 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:15.419499    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podStartSLOduration=2.419480196 podStartE2EDuration="2.419480196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:15.418766296 +0000 UTC m=+3853.468797480" watchObservedRunningTime="2026-11-03 15:16:15.419480196 +0000 UTC m=+3853.469511383"
Nov 03 15:16:15.473729 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:15.473681    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podStartSLOduration=2.4736584649999998 podStartE2EDuration="2.473658465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:15.472685486 +0000 UTC m=+3853.522716685" watchObservedRunningTime="2026-11-03 15:16:15.473658465 +0000 UTC m=+3853.523689642"
Nov 03 15:16:16.133255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:16.132561    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:16.138253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:16.135834    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:16.138253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:16.135890    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:16.147449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:16.147406    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:16.148660 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:16.148636    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:16.148856 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:16.148830    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:17.134637 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:17.134598    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:17.135647 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:17.135613    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:17.147603 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:17.147576    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:17.147807 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:17.147783    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:18.137061 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:18.136616    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:18.137061 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:18.136679    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:18.148755 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:18.148582    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:18.148755 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:18.148634    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:19.131696 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:19.131656    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:19.132132 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:19.132100    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:19.149829 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:19.149661    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:19.149829 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:19.149722    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:20.138704 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:20.138662    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:20.139005 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:20.138974    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:20.149622 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:20.149591    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:20.149851 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:20.149828    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:21.131403 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:21.131362    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:21.131627 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:21.131412    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:21.147678 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:21.147653    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:21.147832 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:21.147691    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:22.132088 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:22.132049    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused" start-of-body=
Nov 03 15:16:22.132502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:22.132117    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": dial tcp 10.133.1.20:15021: connect: connection refused"
Nov 03 15:16:22.147443 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:22.147416    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Nov 03 15:16:22.147509 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:22.147467    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Nov 03 15:16:23.997557 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:23.997478    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:23.998366 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:23.998076    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:23.999021 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:23.998998    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4"
Nov 03 15:16:24.035305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:24.035277    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:24.035832 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:24.035809    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:24.036622 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:24.036598    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6"
Nov 03 15:16:27.985456 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:16:27.985423    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:16:28.877696 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.877663    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"]
Nov 03 15:16:28.881509 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.881490    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.903585 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.903547    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"]
Nov 03 15:16:28.972264 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972215    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972427 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972281    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972427 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972388    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972519 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972426    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972558 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972519    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972558 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972537    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972556    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972619    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:28.972697 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:28.972643    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxph\" (UniqueName: \"kubernetes.io/projected/1cca3427-0571-4cd1-82be-b1bbaaefc72f-kube-api-access-cgxph\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.073755 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073723    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073768    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073807    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073821    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073840    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073857    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.073956    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxph\" (UniqueName: \"kubernetes.io/projected/1cca3427-0571-4cd1-82be-b1bbaaefc72f-kube-api-access-cgxph\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074028    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074256 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074082    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074748 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074348    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074748 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074450    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074748 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074485    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074748 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074537    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.074748 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.074658    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.076251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.076197    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.076456 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.076439    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.094309 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.094285    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cca3427-0571-4cd1-82be-b1bbaaefc72f-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.094778 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.094763    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxph\" (UniqueName: \"kubernetes.io/projected/1cca3427-0571-4cd1-82be-b1bbaaefc72f-kube-api-access-cgxph\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-c8dts\" (UID: \"1cca3427-0571-4cd1-82be-b1bbaaefc72f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.196298 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.196270    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:29.512471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.512444    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"]
Nov 03 15:16:29.514433 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:16:29.514405    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cca3427_0571_4cd1_82be_b1bbaaefc72f.slice/crio-8391ab5f3726016e942b801950b94e3977577797ff69dc5baaf63d204225bd8b WatchSource:0}: Error finding container 8391ab5f3726016e942b801950b94e3977577797ff69dc5baaf63d204225bd8b: Status 404 returned error can't find the container with id 8391ab5f3726016e942b801950b94e3977577797ff69dc5baaf63d204225bd8b
Nov 03 15:16:29.516672 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.516639    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:29.516740 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.516707    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:29.516780 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:29.516745    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:16:30.433565 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:30.433532    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" event={"ID":"1cca3427-0571-4cd1-82be-b1bbaaefc72f","Type":"ContainerStarted","Data":"e4458f9bfa950bb82942d53a2f2395017b6658606afe6d1dd05b2fbb8c45a147"}
Nov 03 15:16:30.433992 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:30.433586    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" event={"ID":"1cca3427-0571-4cd1-82be-b1bbaaefc72f","Type":"ContainerStarted","Data":"8391ab5f3726016e942b801950b94e3977577797ff69dc5baaf63d204225bd8b"}
Nov 03 15:16:30.492433 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:30.492384    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podStartSLOduration=2.492371082 podStartE2EDuration="2.492371082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:16:30.490974784 +0000 UTC m=+3868.541005967" watchObservedRunningTime="2026-11-03 15:16:30.492371082 +0000 UTC m=+3868.542402266"
Nov 03 15:16:31.197116 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:31.197083    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:31.198537 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:31.198514    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:31.198660 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:31.198580    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:32.196785 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:32.196744    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:32.197188 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:32.196796    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:33.197361 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:33.197324    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:33.197772 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:33.197393    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:34.196664 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:34.196624    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:34.196863 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:34.196677    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:35.196990 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:35.196953    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:35.197398 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:35.197014    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:36.197441 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:36.197407    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:36.197825 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:36.197468    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:37.197379 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:37.197340    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused" start-of-body=
Nov 03 15:16:37.197597 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:37.197411    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": dial tcp 10.133.1.22:15021: connect: connection refused"
Nov 03 15:16:38.960482 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:38.960447    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:38.960974 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:38.960871    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:38.961610 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:16:38.961589    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts"
Nov 03 15:16:57.985767 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:16:57.985672    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:17:04.178614 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:04.178579    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:17:05.627926 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:05.627896    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:17:25.383010 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.382975    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"]
Nov 03 15:17:25.387139 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.387118    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.407359 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.407331    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"]
Nov 03 15:17:25.416731 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.416708    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:17:25.416855 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.416727    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:17:25.416855 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.416739    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:17:25.416855 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.416802    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:17:25.543871 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.543837    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.544049 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.543961    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rfl\" (UniqueName: \"kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.544049 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.543988    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.644839 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.644757    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.645005 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.644850    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rfl\" (UniqueName: \"kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.645005 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.644877    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.645420 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.645399    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.647136 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.647114    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.665958 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.665936    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rfl\" (UniqueName: \"kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl\") pod \"maas-api-revoke-sub-1ef70824-w2qbz\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.699733 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.699714    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:25.997140 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:25.997116    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"]
Nov 03 15:17:25.999259 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:17:25.999210    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7fbd1fc_3c5f_4841_a86e_0958ecf0bad7.slice/crio-c750cd17f5386c0723e0b95ac58e704453c6fb87b3eee0fb7417550117860bef WatchSource:0}: Error finding container c750cd17f5386c0723e0b95ac58e704453c6fb87b3eee0fb7417550117860bef: Status 404 returned error can't find the container with id c750cd17f5386c0723e0b95ac58e704453c6fb87b3eee0fb7417550117860bef
Nov 03 15:17:26.698320 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:26.698280    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz" event={"ID":"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7","Type":"ContainerStarted","Data":"97a59836b657be1d37ed5700a51819eab204ff4d1347d33473b8f65be08127b3"}
Nov 03 15:17:26.698691 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:26.698338    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz" event={"ID":"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7","Type":"ContainerStarted","Data":"c750cd17f5386c0723e0b95ac58e704453c6fb87b3eee0fb7417550117860bef"}
Nov 03 15:17:26.740880 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:26.740838    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz" podStartSLOduration=1.740826443 podStartE2EDuration="1.740826443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:26.739219923 +0000 UTC m=+3924.789251106" watchObservedRunningTime="2026-11-03 15:17:26.740826443 +0000 UTC m=+3924.790857662"
Nov 03 15:17:27.704367 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:27.704332    6481 generic.go:356] "Generic (PLEG): container finished" podID="f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" containerID="97a59836b657be1d37ed5700a51819eab204ff4d1347d33473b8f65be08127b3" exitCode=0
Nov 03 15:17:27.704793 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:27.704422    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz" event={"ID":"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7","Type":"ContainerDied","Data":"97a59836b657be1d37ed5700a51819eab204ff4d1347d33473b8f65be08127b3"}
Nov 03 15:17:27.985417 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:17:27.985330    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:17:28.843798 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.843777    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:28.975809 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.975739    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca\") pod \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") "
Nov 03 15:17:28.975956 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.975851    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca\") pod \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") "
Nov 03 15:17:28.975956 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.975943    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl\" (UniqueName: \"kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl\") pod \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\" (UID: \"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7\") "
Nov 03 15:17:28.976097 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.976071    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca" pod "f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" (UID: "f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:17:28.976274 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.976256    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:17:28.977893 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.977859    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca" pod "f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" (UID: "f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:17:28.977994 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:28.977916    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl" pod "f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" (UID: "f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7"). InnerVolumeSpecName "kube-api-access-r4rfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:17:29.077177 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:29.077142    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:17:29.077177 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:29.077172    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4rfl\" (UniqueName: \"kubernetes.io/projected/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7-kube-api-access-r4rfl\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:17:29.715066 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:29.715023    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz" event={"ID":"f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7","Type":"ContainerDied","Data":"c750cd17f5386c0723e0b95ac58e704453c6fb87b3eee0fb7417550117860bef"}
Nov 03 15:17:29.715272 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:29.715103    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c750cd17f5386c0723e0b95ac58e704453c6fb87b3eee0fb7417550117860bef"
Nov 03 15:17:29.715272 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:29.715045    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"
Nov 03 15:17:35.914090 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:35.914059    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:17:36.857927 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:36.857890    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:17:40.010378 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:40.010334    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:17:40.011043 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:40.011010    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:17:40.039422 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:40.039387    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:17:40.039685 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:40.039657    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:17:40.080388 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:40.080357    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:17:40.080641 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:40.080613    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:17:43.125312 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:43.125269    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:17:57.985323 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:17:57.985284    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:17:59.147476 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.147439    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"]
Nov 03 15:17:59.147900 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.147884    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" containerName="revoke-keys"
Nov 03 15:17:59.147946 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.147901    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" containerName="revoke-keys"
Nov 03 15:17:59.147998 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.147989    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" containerName="revoke-keys"
Nov 03 15:17:59.151042 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.151021    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.179565 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.179536    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"]
Nov 03 15:17:59.307422 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307394    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307422 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307425    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307446    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307583    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307652    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307695 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307682    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307911 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307794    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307911 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307822    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.307911 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.307858    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54kcp\" (UniqueName: \"kubernetes.io/projected/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-kube-api-access-54kcp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409016 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.408990    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409021    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409043    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409107    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409159    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409458 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409187    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409458 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409265    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409458 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409289    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409458 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409319    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54kcp\" (UniqueName: \"kubernetes.io/projected/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-kube-api-access-54kcp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409706 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409548    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409706 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409675    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409819 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409786    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409889 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409837    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.409955 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.409933    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.411408 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.411389    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.411532 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.411514    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.430667 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.430647    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.430778 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.430673    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54kcp\" (UniqueName: \"kubernetes.io/projected/8c1222d6-eef5-45c9-88ac-84c8b86a60d3-kube-api-access-54kcp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wxgc4\" (UID: \"8c1222d6-eef5-45c9-88ac-84c8b86a60d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.465665 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.465638    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:17:59.782901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.782875    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"]
Nov 03 15:17:59.784208 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:17:59.784183    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1222d6_eef5_45c9_88ac_84c8b86a60d3.slice/crio-1dba30d1cca0d90c3d34d444542eb015ae0a09f7efe1114ebd06a59c99f0706c WatchSource:0}: Error finding container 1dba30d1cca0d90c3d34d444542eb015ae0a09f7efe1114ebd06a59c99f0706c: Status 404 returned error can't find the container with id 1dba30d1cca0d90c3d34d444542eb015ae0a09f7efe1114ebd06a59c99f0706c
Nov 03 15:17:59.786359 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.786326    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:17:59.786456 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.786391    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:17:59.786456 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.786422    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:17:59.876264 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.876219    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" event={"ID":"8c1222d6-eef5-45c9-88ac-84c8b86a60d3","Type":"ContainerStarted","Data":"efb4412d16d5df64175317dc24353313710e7d9a697fccb6a7478f0d9d2c5c5b"}
Nov 03 15:17:59.876444 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.876280    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" event={"ID":"8c1222d6-eef5-45c9-88ac-84c8b86a60d3","Type":"ContainerStarted","Data":"1dba30d1cca0d90c3d34d444542eb015ae0a09f7efe1114ebd06a59c99f0706c"}
Nov 03 15:17:59.920797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:17:59.920753    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podStartSLOduration=0.920739806 podStartE2EDuration="920.739806ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:17:59.920566839 +0000 UTC m=+3957.970598023" watchObservedRunningTime="2026-11-03 15:17:59.920739806 +0000 UTC m=+3957.970770989"
Nov 03 15:18:00.466382 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:00.466347    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:18:00.467819 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:00.467797    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:00.467883 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:00.467841    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:01.466555 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:01.466520    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:01.466982 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:01.466574    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:02.466547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:02.466514    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:02.466727 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:02.466565    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:03.466453 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:03.466417    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:03.466644 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:03.466470    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:04.287291 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:04.287260    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:18:04.466446 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:04.466408    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:04.466633 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:04.466468    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:05.466935 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:05.466901    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:05.467358 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:05.466951    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:06.466815 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:06.466778    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:06.467045 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:06.466839    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:07.466662 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:07.466622    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Nov 03 15:18:07.466848 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:07.466686    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Nov 03 15:18:08.913046 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:08.913016    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:18:08.913571 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:08.913358    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:18:08.914166 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:08.914144    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4"
Nov 03 15:18:18.235769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:18.235659    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:18:27.985859 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:18:27.985824    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:18:53.342716 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:53.342684    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:18:53.558014 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:18:53.557975    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:18:57.986147 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:18:57.986109    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:19:04.378862 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:04.378827    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:06.465407 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.465374    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:06.900908 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.900831    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"]
Nov 03 15:19:06.904718 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.904699    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.942214 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.942189    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-7b218f2f-openshift-default-dockercfg-j9cpn\""
Nov 03 15:19:06.955881 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.955853    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"]
Nov 03 15:19:06.964442 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964416    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964559 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964454    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964559 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964475    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964549    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964607    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964702 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964635    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5ln2\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964740 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964720    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964780 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964754    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:06.964879 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:06.964863    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.065538 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.065507    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.065779 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.065756    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.065899 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.065809    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.065899 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.065880    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.065998 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.065949    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.065998 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.065989    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066018    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066052    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066079    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066170 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066105    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5ln2\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066485 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066324    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066485 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066384    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066485 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066461    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.066864 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.066844    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.068429 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.068407    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.068524 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.068407    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.086124 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.086105    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.086301 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.086279    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5ln2\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2\") pod \"e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") " pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.109199 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.109175    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:07.218956 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.218920    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:07.314468 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.314423    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:07.536557 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.536535    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"]
Nov 03 15:19:07.538047 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:19:07.538022    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc610e8f7_f1ec_4993_a124_acc62799a079.slice/crio-3420d394792be335608287118b9f6d34b00db94fa20ad544257c2649c1042841 WatchSource:0}: Error finding container 3420d394792be335608287118b9f6d34b00db94fa20ad544257c2649c1042841: Status 404 returned error can't find the container with id 3420d394792be335608287118b9f6d34b00db94fa20ad544257c2649c1042841
Nov 03 15:19:07.540309 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.540278    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:19:07.540376 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.540340    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:19:07.540376 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:07.540371    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:19:08.197100 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:08.197064    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" event={"ID":"c610e8f7-f1ec-4993-a124-acc62799a079","Type":"ContainerStarted","Data":"f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"}
Nov 03 15:19:08.197319 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:08.197116    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" event={"ID":"c610e8f7-f1ec-4993-a124-acc62799a079","Type":"ContainerStarted","Data":"3420d394792be335608287118b9f6d34b00db94fa20ad544257c2649c1042841"}
Nov 03 15:19:08.219314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:08.219289    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:08.224209 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:08.224191    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:08.256834 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:08.256795    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podStartSLOduration=2.256780567 podStartE2EDuration="2.256780567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:19:08.256216954 +0000 UTC m=+4026.306248137" watchObservedRunningTime="2026-11-03 15:19:08.256780567 +0000 UTC m=+4026.306811794"
Nov 03 15:19:09.202018 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:09.201990    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:09.202999 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:09.202980    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:10.797168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:10.797138    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:11.033318 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:11.033288    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:12.642574 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:12.642541    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:12.857432 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:12.857397    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:14.698911 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:14.698880    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:16.599730 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:16.599693    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:16.666620 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:16.666582    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:19:18.575443 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:18.575407    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:20.479693 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:20.479660    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:22.422257 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:22.419633    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:24.491983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:24.491944    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:25.243359 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:25.243327    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:19:25.243564 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:25.243383    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:19:25.607509 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:25.607433    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:26.426020 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:26.425988    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:26.634443 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:26.634375    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:27.009967 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.009938    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:27.302625 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.302544    6481 generic.go:356] "Generic (PLEG): container finished" podID="c610e8f7-f1ec-4993-a124-acc62799a079" containerID="f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609" exitCode=0
Nov 03 15:19:27.302625 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.302612    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" event={"ID":"c610e8f7-f1ec-4993-a124-acc62799a079","Type":"ContainerDied","Data":"f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"}
Nov 03 15:19:27.302971 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.302957    6481 scope.go:122] "RemoveContainer" containerID="f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"
Nov 03 15:19:27.304121 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.304097    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:19:27.304185 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.304159    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:19:27.304222 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:27.304190    6481 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 03 15:19:27.986330 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:19:27.986299    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:19:28.332809 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:28.332725    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" event={"ID":"c610e8f7-f1ec-4993-a124-acc62799a079","Type":"ContainerStarted","Data":"42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7"}
Nov 03 15:19:28.471125 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:28.471097    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:29.221993 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:29.221578    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:29.221993 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:29.221695    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Nov 03 15:19:29.221993 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:29.221739    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Nov 03 15:19:30.219763 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:30.219714    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Nov 03 15:19:30.220091 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:30.220045    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Nov 03 15:19:30.370858 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:30.370819    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:31.221113 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:31.220637    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Nov 03 15:19:31.221113 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:31.220719    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Nov 03 15:19:32.225610 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:32.225570    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Nov 03 15:19:32.226134 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:32.226102    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Nov 03 15:19:32.289546 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:32.289516    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:33.220190 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:33.220148    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Nov 03 15:19:33.220403 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:33.220254    6481 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Nov 03 15:19:34.174994 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:34.174950    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:34.222693 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:34.222667    6481 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:34.223080 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:34.223060    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:34.223660 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:34.223638    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:19:34.382927 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:34.382893    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:36.076428 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:36.076399    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:37.979214 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:37.979183    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:39.882712 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:39.882679    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:41.704012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:41.703981    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:43.583432 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:43.583348    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:45.477923 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:45.477888    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:47.396584 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:47.396549    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:49.277945 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:49.277913    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:51.184968 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:51.184936    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:53.068288 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:53.068248    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:54.986265 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:54.986195    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:56.871956 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:56.871907    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:19:57.986312 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:19:57.986275    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:19:58.803859 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:19:58.803823    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:00.686108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:00.686072    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:02.566980 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:02.566930    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:04.482927 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:04.482896    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:04.687679 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:04.687648    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:05.136790 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:05.136753    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:06.481578 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:06.481545    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:08.480239 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:08.480183    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:10.474283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:10.474247    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:12.468371 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:12.468334    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:14.393283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:14.393224    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:16.282839 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:16.282809    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:18.213142 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:18.213112    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:20.092754 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:20.092723    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:22.074709 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:22.074668    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:24.066241 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:24.066197    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:25.143557 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:25.143522    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:25.368579 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:25.368543    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:26.091661 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:26.091632    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:26.367609 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:26.367521    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:27.986306 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:20:27.986273    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:20:28.006416 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:28.006386    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:29.914346 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:29.914314    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:31.869090 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:31.869062    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:33.810706 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:33.810675    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:35.698885 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:35.698854    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:37.675430 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:37.675399    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:39.682209 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:39.682177    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:41.696051 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:41.696018    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:43.700776 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:43.700749    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:45.680026 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:45.679991    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:47.675940 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:47.675905    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:48.050378 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:48.050296    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"]
Nov 03 15:20:48.065401 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:48.065373    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ead6fc6-68mb2"]
Nov 03 15:20:49.615107 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:49.615073    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:50.065319 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:50.065288    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b9606c-241e-4443-b9d1-93e3143fa707" path="/var/lib/kubelet/pods/60b9606c-241e-4443-b9d1-93e3143fa707/volumes"
Nov 03 15:20:51.596650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:51.596621    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:52.928443 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:52.928404    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:53.149980 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:53.149951    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:53.594035 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:53.594004    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:55.514421 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:55.514389    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:57.525220 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:57.525184    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:57.886652 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:57.886570    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:20:57.986133 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:20:57.986099    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:20:59.502012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:20:59.501979    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:01.611825 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:01.611794    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:03.578325 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:03.578293    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:04.640570 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:04.640533    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:05.746166 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:05.746128    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:06.058854 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:06.058772    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:06.353328 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:06.353245    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:06.599921 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:06.599894    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:07.897874 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:07.897831    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:09.790441 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:09.790405    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:12.119603 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:12.119560    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:21:14.159160 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:14.159082    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:14.382161 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:14.382129    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:14.999345 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:14.999309    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:15.379485 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:15.379396    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"]
Nov 03 15:21:15.379901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:15.379629    6481 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" containerID="cri-o://42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7" gracePeriod=30
Nov 03 15:21:15.509554 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:15.509512    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:19.224775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:19.224743    6481 patch_prober.go:29] interesting pod/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 15:21:19.225175 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:19.224795    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 15:21:19.583244 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:19.583142    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:21:20.720135 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.720101    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:21:20.826984 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.826947    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827354 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827017    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827354 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827072    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827354 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827116    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827354 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827166    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827354 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827288    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827697 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827369    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827697 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827415    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.827697 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.827452    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert\") pod \"c610e8f7-f1ec-4993-a124-acc62799a079\" (UID: \"c610e8f7-f1ec-4993-a124-acc62799a079\") "
Nov 03 15:21:20.828284 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.828258    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:21:20.828898 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.828842    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:21:20.829111 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.829074    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:21:20.829295 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.829264    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:21:20.829541 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.829508    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:21:20.830206 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.830185    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:21:20.830806 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.830767    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 15:21:20.831535 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.831507    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "kube-api-access-k5ln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:21:20.831766 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.831739    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy" pod "c610e8f7-f1ec-4993-a124-acc62799a079" (UID: "c610e8f7-f1ec-4993-a124-acc62799a079"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 15:21:20.887618 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.887582    6481 generic.go:356] "Generic (PLEG): container finished" podID="c610e8f7-f1ec-4993-a124-acc62799a079" containerID="42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7" exitCode=0
Nov 03 15:21:20.887822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.887638    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"
Nov 03 15:21:20.887822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.887657    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" event={"ID":"c610e8f7-f1ec-4993-a124-acc62799a079","Type":"ContainerDied","Data":"42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7"}
Nov 03 15:21:20.887822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.887760    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76" event={"ID":"c610e8f7-f1ec-4993-a124-acc62799a079","Type":"ContainerDied","Data":"3420d394792be335608287118b9f6d34b00db94fa20ad544257c2649c1042841"}
Nov 03 15:21:20.887822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.887794    6481 scope.go:122] "RemoveContainer" containerID="42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7"
Nov 03 15:21:20.909414 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.909392    6481 scope.go:122] "RemoveContainer" containerID="f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"
Nov 03 15:21:20.922988 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.922964    6481 scope.go:122] "RemoveContainer" containerID="42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7"
Nov 03 15:21:20.923385 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:21:20.923346    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7\": container with ID starting with 42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7 not found: ID does not exist" containerID="42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7"
Nov 03 15:21:20.923507 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.923388    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7"} err="failed to get container status \"42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7\": rpc error: code = NotFound desc = could not find container \"42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7\": container with ID starting with 42027f1668fb500bcb216ccb8f11e10d1e76b52e99716ed6fe501152266d22a7 not found: ID does not exist"
Nov 03 15:21:20.923507 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.923415    6481 scope.go:122] "RemoveContainer" containerID="f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"
Nov 03 15:21:20.923739 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:21:20.923705    6481 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609\": container with ID starting with f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609 not found: ID does not exist" containerID="f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"
Nov 03 15:21:20.923861 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.923746    6481 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609"} err="failed to get container status \"f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609\": rpc error: code = NotFound desc = could not find container \"f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609\": container with ID starting with f7bd8cd860e369e52653729e9cf3af2c8395d33d0004e24ec47a7526f6375609 not found: ID does not exist"
Nov 03 15:21:20.929525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929493    6481 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-certs\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929528    6481 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c610e8f7-f1ec-4993-a124-acc62799a079-istiod-ca-cert\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929542    6481 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-envoy\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929558    6481 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-istio-token\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929571    6481 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-workload-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929585    6481 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-istio-data\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929599    6481 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c610e8f7-f1ec-4993-a124-acc62799a079-istio-podinfo\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929612    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5ln2\" (UniqueName: \"kubernetes.io/projected/c610e8f7-f1ec-4993-a124-acc62799a079-kube-api-access-k5ln2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.929737 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.929627    6481 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c610e8f7-f1ec-4993-a124-acc62799a079-credential-socket\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:20.950450 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.950382    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"]
Nov 03 15:21:20.955701 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:20.955674    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-7b218f2f-openshift-default-67d75c69f7-69r76"]
Nov 03 15:21:22.098769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:22.098726    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" path="/var/lib/kubelet/pods/c610e8f7-f1ec-4993-a124-acc62799a079/volumes"
Nov 03 15:21:28.028319 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:21:28.028252    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:21:30.325687 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.325657    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"]
Nov 03 15:21:30.326108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.326082    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy"
Nov 03 15:21:30.326108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.326095    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy"
Nov 03 15:21:30.326190 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.326114    6481 cpu_manager.go:414] "containerMap: removing container" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy"
Nov 03 15:21:30.326190 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.326120    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy"
Nov 03 15:21:30.326293 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.326192    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy"
Nov 03 15:21:30.326293 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.326202    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="c610e8f7-f1ec-4993-a124-acc62799a079" containerName="istio-proxy"
Nov 03 15:21:30.330542 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.330521    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.350373 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.350347    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"]
Nov 03 15:21:30.360512 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.360488    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:21:30.360663 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.360545    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:21:30.360663 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.360564    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:21:30.360663 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.360585    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:21:30.463252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.463191    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.463431 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.463265    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.463431 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.463354    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnp8q\" (UniqueName: \"kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.564900 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.564855    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.565091 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.564921    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.565091 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.564958    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnp8q\" (UniqueName: \"kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.565535 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.565511    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.567310 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.567290    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.588084 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.588028    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnp8q\" (UniqueName: \"kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q\") pod \"maas-api-revoke-sub-81f076c9-dwccc\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.644760 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.644733    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:30.784400 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.784369    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:21:30.943582 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.943560    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"]
Nov 03 15:21:30.945263 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:21:30.945215    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb70daab9_b0f4_4389_9cd4_40e8cb05bbd1.slice/crio-adda9c2aefa3d3174ab9e2a3a8dc6938c5e53dad27d8c3cddcebe4311d5c5e79 WatchSource:0}: Error finding container adda9c2aefa3d3174ab9e2a3a8dc6938c5e53dad27d8c3cddcebe4311d5c5e79: Status 404 returned error can't find the container with id adda9c2aefa3d3174ab9e2a3a8dc6938c5e53dad27d8c3cddcebe4311d5c5e79
Nov 03 15:21:30.994334 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:30.994310    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc" event={"ID":"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1","Type":"ContainerStarted","Data":"adda9c2aefa3d3174ab9e2a3a8dc6938c5e53dad27d8c3cddcebe4311d5c5e79"}
Nov 03 15:21:32.007814 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:32.007780    6481 generic.go:356] "Generic (PLEG): container finished" podID="b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" containerID="4431d0237aa44eb21bea1a6852b30a3bb70a00bb3ec7eb2d8093d90ee74cce8a" exitCode=0
Nov 03 15:21:32.008267 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:32.007870    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc" event={"ID":"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1","Type":"ContainerDied","Data":"4431d0237aa44eb21bea1a6852b30a3bb70a00bb3ec7eb2d8093d90ee74cce8a"}
Nov 03 15:21:33.153924 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.153900    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:33.290640 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.290566    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q\" (UniqueName: \"kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q\") pod \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") "
Nov 03 15:21:33.290640 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.290606    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca\") pod \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") "
Nov 03 15:21:33.290640 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.290633    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca\") pod \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\" (UID: \"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1\") "
Nov 03 15:21:33.291030 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.290999    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca" pod "b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" (UID: "b70daab9-b0f4-4389-9cd4-40e8cb05bbd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:21:33.292673 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.292642    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca" pod "b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" (UID: "b70daab9-b0f4-4389-9cd4-40e8cb05bbd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:21:33.292786 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.292750    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q" pod "b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" (UID: "b70daab9-b0f4-4389-9cd4-40e8cb05bbd1"). InnerVolumeSpecName "kube-api-access-cnp8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:21:33.392168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.392140    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnp8q\" (UniqueName: \"kubernetes.io/projected/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-kube-api-access-cnp8q\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:33.392168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.392163    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:33.392168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:33.392173    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:34.018207 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:34.018173    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc" event={"ID":"b70daab9-b0f4-4389-9cd4-40e8cb05bbd1","Type":"ContainerDied","Data":"adda9c2aefa3d3174ab9e2a3a8dc6938c5e53dad27d8c3cddcebe4311d5c5e79"}
Nov 03 15:21:34.018410 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:34.018222    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adda9c2aefa3d3174ab9e2a3a8dc6938c5e53dad27d8c3cddcebe4311d5c5e79"
Nov 03 15:21:34.018410 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:34.018181    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"
Nov 03 15:21:40.753974 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.753934    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"]
Nov 03 15:21:40.754411 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.754369    6481 cpu_manager.go:414] "containerMap: removing container" podUID="b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" containerName="revoke-keys"
Nov 03 15:21:40.754411 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.754381    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" containerName="revoke-keys"
Nov 03 15:21:40.754504 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.754462    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" containerName="revoke-keys"
Nov 03 15:21:40.757305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.757286    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.782434 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.782405    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"]
Nov 03 15:21:40.784646 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.784624    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:21:40.784797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.784654    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:21:40.784797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.784666    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:21:40.784797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.784697    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:21:40.865686 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.865656    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.865859 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.865750    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.865859 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.865793    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.967124 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.967081    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.967320 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.967142    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.967320 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.967299    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.967956 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.967937    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.969466 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.969448    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:40.988538 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:40.988510    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk\") pod \"maas-api-revoke-sub-94f68e24-slzs6\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:41.070499 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:41.070431    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:41.370959 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:41.370930    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"]
Nov 03 15:21:41.372737 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:21:41.372713    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e291e8_9e32_496f_bb17_4413d100335c.slice/crio-9f9535c38cf4d336d33c04eef48fc6dc698ef6c7fcca8f990e9ac9f90a6bc17b WatchSource:0}: Error finding container 9f9535c38cf4d336d33c04eef48fc6dc698ef6c7fcca8f990e9ac9f90a6bc17b: Status 404 returned error can't find the container with id 9f9535c38cf4d336d33c04eef48fc6dc698ef6c7fcca8f990e9ac9f90a6bc17b
Nov 03 15:21:42.057925 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:42.057893    6481 generic.go:356] "Generic (PLEG): container finished" podID="e7e291e8-9e32-496f-bb17-4413d100335c" containerID="410a6d3281e191de9fb8981e9e760bebb75fc6dcdce2bbe5773142c31bae7a61" exitCode=0
Nov 03 15:21:42.058283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:42.057973    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6" event={"ID":"e7e291e8-9e32-496f-bb17-4413d100335c","Type":"ContainerDied","Data":"410a6d3281e191de9fb8981e9e760bebb75fc6dcdce2bbe5773142c31bae7a61"}
Nov 03 15:21:42.058283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:42.058023    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6" event={"ID":"e7e291e8-9e32-496f-bb17-4413d100335c","Type":"ContainerStarted","Data":"9f9535c38cf4d336d33c04eef48fc6dc698ef6c7fcca8f990e9ac9f90a6bc17b"}
Nov 03 15:21:43.195105 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.195080    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:43.289531 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.289497    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca\") pod \"e7e291e8-9e32-496f-bb17-4413d100335c\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") "
Nov 03 15:21:43.289711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.289575    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca\") pod \"e7e291e8-9e32-496f-bb17-4413d100335c\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") "
Nov 03 15:21:43.289711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.289639    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk\") pod \"e7e291e8-9e32-496f-bb17-4413d100335c\" (UID: \"e7e291e8-9e32-496f-bb17-4413d100335c\") "
Nov 03 15:21:43.289930 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.289893    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca" pod "e7e291e8-9e32-496f-bb17-4413d100335c" (UID: "e7e291e8-9e32-496f-bb17-4413d100335c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:21:43.290069 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.289962    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:43.291714 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.291692    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca" pod "e7e291e8-9e32-496f-bb17-4413d100335c" (UID: "e7e291e8-9e32-496f-bb17-4413d100335c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:21:43.291785 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.291709    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk" pod "e7e291e8-9e32-496f-bb17-4413d100335c" (UID: "e7e291e8-9e32-496f-bb17-4413d100335c"). InnerVolumeSpecName "kube-api-access-88mtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:21:43.390528 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.390469    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88mtk\" (UniqueName: \"kubernetes.io/projected/e7e291e8-9e32-496f-bb17-4413d100335c-kube-api-access-88mtk\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:43.390528 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:43.390495    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7e291e8-9e32-496f-bb17-4413d100335c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:21:44.068517 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:44.068488    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"
Nov 03 15:21:44.068692 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:44.068512    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6" event={"ID":"e7e291e8-9e32-496f-bb17-4413d100335c","Type":"ContainerDied","Data":"9f9535c38cf4d336d33c04eef48fc6dc698ef6c7fcca8f990e9ac9f90a6bc17b"}
Nov 03 15:21:44.068692 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:44.068557    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f9535c38cf4d336d33c04eef48fc6dc698ef6c7fcca8f990e9ac9f90a6bc17b"
Nov 03 15:21:52.066639 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:52.066607    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:21:54.955398 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:54.955359    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:21:54.955936 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:54.955428    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:21:56.872121 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:21:56.872087    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:21:58.000340 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:21:58.000291    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:22:04.757264 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:04.757215    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:22:06.457650 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.457614    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"]
Nov 03 15:22:06.458266 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.458245    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e7e291e8-9e32-496f-bb17-4413d100335c" containerName="revoke-keys"
Nov 03 15:22:06.458368 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.458268    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e291e8-9e32-496f-bb17-4413d100335c" containerName="revoke-keys"
Nov 03 15:22:06.458432 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.458411    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e291e8-9e32-496f-bb17-4413d100335c" containerName="revoke-keys"
Nov 03 15:22:06.462959 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.462936    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.483475 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.483446    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"]
Nov 03 15:22:06.491987 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.491956    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:22:06.492440 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.492299    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:22:06.492440 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.492370    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:22:06.492689 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.492391    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:22:06.570403 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.570369    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.570589 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.570517    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx2zd\" (UniqueName: \"kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.570589 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.570562    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.672080 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.672041    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx2zd\" (UniqueName: \"kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.672314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.672095    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.672314 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.672173    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.672923 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.672902    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.674525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.674501    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.693978 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.693958    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx2zd\" (UniqueName: \"kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd\") pod \"maas-api-revoke-sub-3745d2b7-w8t5j\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:06.775665 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:06.775603    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:07.080381 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:07.080357    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"]
Nov 03 15:22:07.082739 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:22:07.082712    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aefa24_dc79_4d8d_8c62_c79e1829145a.slice/crio-31211d4d0b281f266eb1ce2079bbda31752a5854ef1ef240ce7ddc74ef326b8d WatchSource:0}: Error finding container 31211d4d0b281f266eb1ce2079bbda31752a5854ef1ef240ce7ddc74ef326b8d: Status 404 returned error can't find the container with id 31211d4d0b281f266eb1ce2079bbda31752a5854ef1ef240ce7ddc74ef326b8d
Nov 03 15:22:07.228565 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:07.228522    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j" event={"ID":"58aefa24-dc79-4d8d-8c62-c79e1829145a","Type":"ContainerStarted","Data":"31211d4d0b281f266eb1ce2079bbda31752a5854ef1ef240ce7ddc74ef326b8d"}
Nov 03 15:22:08.237340 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:08.237304    6481 generic.go:356] "Generic (PLEG): container finished" podID="58aefa24-dc79-4d8d-8c62-c79e1829145a" containerID="78b1f98eeb091d4a7b9fac769aab79ba8283f7fb9e7a7097012999eb7d378207" exitCode=0
Nov 03 15:22:08.237765 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:08.237388    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j" event={"ID":"58aefa24-dc79-4d8d-8c62-c79e1829145a","Type":"ContainerDied","Data":"78b1f98eeb091d4a7b9fac769aab79ba8283f7fb9e7a7097012999eb7d378207"}
Nov 03 15:22:09.375896 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.375869    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:09.384835 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.384810    6481 scope.go:122] "RemoveContainer" containerID="2c4274f4be73ee41da446a8f122a3699ccfc6e3d9b17f44d16dc4daed75aa75d"
Nov 03 15:22:09.500193 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.500108    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca\") pod \"58aefa24-dc79-4d8d-8c62-c79e1829145a\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") "
Nov 03 15:22:09.500193 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.500192    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca\") pod \"58aefa24-dc79-4d8d-8c62-c79e1829145a\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") "
Nov 03 15:22:09.500547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.500270    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd\" (UniqueName: \"kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd\") pod \"58aefa24-dc79-4d8d-8c62-c79e1829145a\" (UID: \"58aefa24-dc79-4d8d-8c62-c79e1829145a\") "
Nov 03 15:22:09.500547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.500476    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca" pod "58aefa24-dc79-4d8d-8c62-c79e1829145a" (UID: "58aefa24-dc79-4d8d-8c62-c79e1829145a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:22:09.500635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.500582    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:09.502302 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.502278    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca" pod "58aefa24-dc79-4d8d-8c62-c79e1829145a" (UID: "58aefa24-dc79-4d8d-8c62-c79e1829145a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:22:09.502391 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.502351    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd" pod "58aefa24-dc79-4d8d-8c62-c79e1829145a" (UID: "58aefa24-dc79-4d8d-8c62-c79e1829145a"). InnerVolumeSpecName "kube-api-access-mx2zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:22:09.601638 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.601609    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aefa24-dc79-4d8d-8c62-c79e1829145a-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:09.601638 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:09.601636    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx2zd\" (UniqueName: \"kubernetes.io/projected/58aefa24-dc79-4d8d-8c62-c79e1829145a-kube-api-access-mx2zd\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:10.249907 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:10.249879    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"
Nov 03 15:22:10.250103 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:10.249884    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j" event={"ID":"58aefa24-dc79-4d8d-8c62-c79e1829145a","Type":"ContainerDied","Data":"31211d4d0b281f266eb1ce2079bbda31752a5854ef1ef240ce7ddc74ef326b8d"}
Nov 03 15:22:10.250103 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:10.249999    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31211d4d0b281f266eb1ce2079bbda31752a5854ef1ef240ce7ddc74ef326b8d"
Nov 03 15:22:17.867484 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:17.867451    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:22:22.771950 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:22.771912    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:22:27.985122 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:22:27.985085    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:22:28.301069 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:28.300990    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:22:29.049160 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:29.049110    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"]
Nov 03 15:22:29.060691 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:29.060655    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef70824-w2qbz"]
Nov 03 15:22:30.087414 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:30.087380    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7" path="/var/lib/kubelet/pods/f7fbd1fc-3c5f-4841-a86e-0958ecf0bad7/volumes"
Nov 03 15:22:39.792209 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:39.792173    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:22:40.002183 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.002132    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:22:40.830196 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.830166    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"]
Nov 03 15:22:40.830633 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.830617    6481 cpu_manager.go:414] "containerMap: removing container" podUID="58aefa24-dc79-4d8d-8c62-c79e1829145a" containerName="revoke-keys"
Nov 03 15:22:40.830678 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.830633    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aefa24-dc79-4d8d-8c62-c79e1829145a" containerName="revoke-keys"
Nov 03 15:22:40.830732 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.830723    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aefa24-dc79-4d8d-8c62-c79e1829145a" containerName="revoke-keys"
Nov 03 15:22:40.833780 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.833761    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.858537 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.858511    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.858681 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.858595    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.858760 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.858691    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwddx\" (UniqueName: \"kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.860456 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.860435    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:22:40.860567 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.860518    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:22:40.860800 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.860778    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:22:40.860914 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.860889    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:22:40.863520 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.863501    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"]
Nov 03 15:22:40.960217 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.960184    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.960386 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.960258    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwddx\" (UniqueName: \"kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.960386 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.960365    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.960925 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.960902    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.962590 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.962567    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:40.980893 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:40.980866    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwddx\" (UniqueName: \"kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx\") pod \"maas-api-revoke-sub-d13d82ee-7pfd5\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:41.148823 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:41.148742    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:41.447980 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:41.447957    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"]
Nov 03 15:22:41.449818 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:22:41.449793    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5164cf_509f_42ce_9fef_5df14a91aebb.slice/crio-d1772b63d734dee2146994b36c60aa3b19df4b6c452fd8ee779e8ed94202129b WatchSource:0}: Error finding container d1772b63d734dee2146994b36c60aa3b19df4b6c452fd8ee779e8ed94202129b: Status 404 returned error can't find the container with id d1772b63d734dee2146994b36c60aa3b19df4b6c452fd8ee779e8ed94202129b
Nov 03 15:22:41.520126 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:41.520096    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5" event={"ID":"ff5164cf-509f-42ce-9fef-5df14a91aebb","Type":"ContainerStarted","Data":"d1772b63d734dee2146994b36c60aa3b19df4b6c452fd8ee779e8ed94202129b"}
Nov 03 15:22:42.526305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:42.526213    6481 generic.go:356] "Generic (PLEG): container finished" podID="ff5164cf-509f-42ce-9fef-5df14a91aebb" containerID="3afa814b1b768560dddd94f02789423e2a57f6d71ed264d8061b403cc1ac0224" exitCode=0
Nov 03 15:22:42.526305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:42.526279    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5" event={"ID":"ff5164cf-509f-42ce-9fef-5df14a91aebb","Type":"ContainerDied","Data":"3afa814b1b768560dddd94f02789423e2a57f6d71ed264d8061b403cc1ac0224"}
Nov 03 15:22:43.661400 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.661373    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:43.687276 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.687251    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca\") pod \"ff5164cf-509f-42ce-9fef-5df14a91aebb\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") "
Nov 03 15:22:43.687445 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.687288    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca\") pod \"ff5164cf-509f-42ce-9fef-5df14a91aebb\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") "
Nov 03 15:22:43.687445 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.687306    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx\" (UniqueName: \"kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx\") pod \"ff5164cf-509f-42ce-9fef-5df14a91aebb\" (UID: \"ff5164cf-509f-42ce-9fef-5df14a91aebb\") "
Nov 03 15:22:43.687629 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.687608    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca" pod "ff5164cf-509f-42ce-9fef-5df14a91aebb" (UID: "ff5164cf-509f-42ce-9fef-5df14a91aebb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:22:43.689349 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.689328    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx" pod "ff5164cf-509f-42ce-9fef-5df14a91aebb" (UID: "ff5164cf-509f-42ce-9fef-5df14a91aebb"). InnerVolumeSpecName "kube-api-access-vwddx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:22:43.689427 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.689359    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca" pod "ff5164cf-509f-42ce-9fef-5df14a91aebb" (UID: "ff5164cf-509f-42ce-9fef-5df14a91aebb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:22:43.788990 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.788927    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:43.788990 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.788953    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff5164cf-509f-42ce-9fef-5df14a91aebb-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:43.788990 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:43.788967    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwddx\" (UniqueName: \"kubernetes.io/projected/ff5164cf-509f-42ce-9fef-5df14a91aebb-kube-api-access-vwddx\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:44.536664 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:44.536634    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5"
Nov 03 15:22:44.536846 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:44.536675    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d13d82ee-7pfd5" event={"ID":"ff5164cf-509f-42ce-9fef-5df14a91aebb","Type":"ContainerDied","Data":"d1772b63d734dee2146994b36c60aa3b19df4b6c452fd8ee779e8ed94202129b"}
Nov 03 15:22:44.536846 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:44.536724    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1772b63d734dee2146994b36c60aa3b19df4b6c452fd8ee779e8ed94202129b"
Nov 03 15:22:51.222467 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.222421    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"]
Nov 03 15:22:51.222977 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.222864    6481 cpu_manager.go:414] "containerMap: removing container" podUID="ff5164cf-509f-42ce-9fef-5df14a91aebb" containerName="revoke-keys"
Nov 03 15:22:51.222977 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.222876    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff5164cf-509f-42ce-9fef-5df14a91aebb" containerName="revoke-keys"
Nov 03 15:22:51.222977 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.222943    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff5164cf-509f-42ce-9fef-5df14a91aebb" containerName="revoke-keys"
Nov 03 15:22:51.232523 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.232490    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.247196 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.247167    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"]
Nov 03 15:22:51.256085 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.256055    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:22:51.256242 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.256097    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:22:51.256242 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.256130    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:22:51.259012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.258995    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:22:51.363507 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.363474    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.363698 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.363632    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.363758 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.363696    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpsk9\" (UniqueName: \"kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.464578 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.464545    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.464725 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.464663    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.464803 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.464723    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpsk9\" (UniqueName: \"kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.465322 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.465301    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.466795 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.466770    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.489029 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.488973    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpsk9\" (UniqueName: \"kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9\") pod \"maas-api-revoke-sub-83bc83d0-wc2w7\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.549430 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.549397    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:51.849402 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:51.849377    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"]
Nov 03 15:22:51.852208 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:22:51.852180    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f37139_c35a_4cbe_ac67_3cc7b25689d5.slice/crio-264f60d061569a6066c9d10f236309ce9e1ef8ad885560a8fc5faf36a4928a99 WatchSource:0}: Error finding container 264f60d061569a6066c9d10f236309ce9e1ef8ad885560a8fc5faf36a4928a99: Status 404 returned error can't find the container with id 264f60d061569a6066c9d10f236309ce9e1ef8ad885560a8fc5faf36a4928a99
Nov 03 15:22:52.182077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:52.182048    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:22:52.583252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:52.582714    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7" event={"ID":"f2f37139-c35a-4cbe-ac67-3cc7b25689d5","Type":"ContainerStarted","Data":"4fc6d6cc025f70db3098b81c3604a7ac2c9a69e9e1d31979448e8a97c91ab776"}
Nov 03 15:22:52.583252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:52.582777    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7" event={"ID":"f2f37139-c35a-4cbe-ac67-3cc7b25689d5","Type":"ContainerStarted","Data":"264f60d061569a6066c9d10f236309ce9e1ef8ad885560a8fc5faf36a4928a99"}
Nov 03 15:22:52.626726 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:52.626660    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7" podStartSLOduration=1.626640107 podStartE2EDuration="1.626640107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:22:52.623383558 +0000 UTC m=+4250.673414788" watchObservedRunningTime="2026-11-03 15:22:52.626640107 +0000 UTC m=+4250.676671292"
Nov 03 15:22:53.588741 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:53.588708    6481 generic.go:356] "Generic (PLEG): container finished" podID="f2f37139-c35a-4cbe-ac67-3cc7b25689d5" containerID="4fc6d6cc025f70db3098b81c3604a7ac2c9a69e9e1d31979448e8a97c91ab776" exitCode=0
Nov 03 15:22:53.589136 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:53.588767    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7" event={"ID":"f2f37139-c35a-4cbe-ac67-3cc7b25689d5","Type":"ContainerDied","Data":"4fc6d6cc025f70db3098b81c3604a7ac2c9a69e9e1d31979448e8a97c91ab776"}
Nov 03 15:22:54.457394 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.457364    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:22:54.734540 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.734513    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:54.798591 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.798555    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca\") pod \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") "
Nov 03 15:22:54.798760 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.798698    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca\") pod \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") "
Nov 03 15:22:54.798833 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.798820    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9\" (UniqueName: \"kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9\") pod \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\" (UID: \"f2f37139-c35a-4cbe-ac67-3cc7b25689d5\") "
Nov 03 15:22:54.799075 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.799046    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca" pod "f2f37139-c35a-4cbe-ac67-3cc7b25689d5" (UID: "f2f37139-c35a-4cbe-ac67-3cc7b25689d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:22:54.799266 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.799246    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:54.800746 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.800718    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca" pod "f2f37139-c35a-4cbe-ac67-3cc7b25689d5" (UID: "f2f37139-c35a-4cbe-ac67-3cc7b25689d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:22:54.800836 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.800800    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9" pod "f2f37139-c35a-4cbe-ac67-3cc7b25689d5" (UID: "f2f37139-c35a-4cbe-ac67-3cc7b25689d5"). InnerVolumeSpecName "kube-api-access-xpsk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:22:54.900472 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.900446    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpsk9\" (UniqueName: \"kubernetes.io/projected/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-kube-api-access-xpsk9\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:54.900472 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:54.900469    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f37139-c35a-4cbe-ac67-3cc7b25689d5-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:22:55.599775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:55.599744    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7"
Nov 03 15:22:55.599775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:55.599763    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83bc83d0-wc2w7" event={"ID":"f2f37139-c35a-4cbe-ac67-3cc7b25689d5","Type":"ContainerDied","Data":"264f60d061569a6066c9d10f236309ce9e1ef8ad885560a8fc5faf36a4928a99"}
Nov 03 15:22:55.600004 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:55.599805    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="264f60d061569a6066c9d10f236309ce9e1ef8ad885560a8fc5faf36a4928a99"
Nov 03 15:22:57.986519 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:22:57.986480    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:22:58.645042 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:22:58.644974    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:23:02.499312 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:02.499268    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:23:04.845480 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:04.845454    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:23:07.204472 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:07.204437    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:23:09.973254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:09.969306    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:23:09.973254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:09.969393    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wxgc4" podUID="8c1222d6-eef5-45c9-88ac-84c8b86a60d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:23:10.009253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:10.008384    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-c8dts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:23:10.009253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:10.008445    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-c8dts" podUID="1cca3427-0571-4cd1-82be-b1bbaaefc72f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:23:10.045384 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:10.045343    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:23:10.045698 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:10.045649    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:23:10.095428 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:10.095315    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:23:10.095428 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:10.095386    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:23:14.079858 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:14.079823    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:23:22.140072 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.140031    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"]
Nov 03 15:23:22.140676 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.140655    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f2f37139-c35a-4cbe-ac67-3cc7b25689d5" containerName="revoke-keys"
Nov 03 15:23:22.140769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.140678    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f37139-c35a-4cbe-ac67-3cc7b25689d5" containerName="revoke-keys"
Nov 03 15:23:22.140824 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.140801    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f37139-c35a-4cbe-ac67-3cc7b25689d5" containerName="revoke-keys"
Nov 03 15:23:22.143950 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.143929    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.170774 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.170746    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:23:22.170932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.170905    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:23:22.170932 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.170910    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:23:22.171089 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.170972    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:23:22.174428 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.174399    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"]
Nov 03 15:23:22.260781 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.260749    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.260971 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.260792    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn24b\" (UniqueName: \"kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.260971 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.260813    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.362145 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.362107    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.362371 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.362175    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn24b\" (UniqueName: \"kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.362371 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.362202    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.362882 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.362864    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.364417 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.364396    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.384047 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.384017    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn24b\" (UniqueName: \"kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b\") pod \"maas-api-revoke-sub-9d8eeeb0-pv2nt\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.459454 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.459433    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:22.756828 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.756798    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"]
Nov 03 15:23:22.758645 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:23:22.758616    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26535a8e_c190_49a8_8818_7866725b4d78.slice/crio-3b06bb1d7b739819bcd12710869a46b33e30d6b6869051bf6afa3c2282a73dca WatchSource:0}: Error finding container 3b06bb1d7b739819bcd12710869a46b33e30d6b6869051bf6afa3c2282a73dca: Status 404 returned error can't find the container with id 3b06bb1d7b739819bcd12710869a46b33e30d6b6869051bf6afa3c2282a73dca
Nov 03 15:23:22.883238 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:22.883199    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt" event={"ID":"26535a8e-c190-49a8-8818-7866725b4d78","Type":"ContainerStarted","Data":"3b06bb1d7b739819bcd12710869a46b33e30d6b6869051bf6afa3c2282a73dca"}
Nov 03 15:23:23.897039 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:23.896958    6481 generic.go:356] "Generic (PLEG): container finished" podID="26535a8e-c190-49a8-8818-7866725b4d78" containerID="1527f3d60be290fbfce80e97cd206b0e3760629299331534038a1acaaaf55159" exitCode=0
Nov 03 15:23:23.897420 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:23.897033    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt" event={"ID":"26535a8e-c190-49a8-8818-7866725b4d78","Type":"ContainerDied","Data":"1527f3d60be290fbfce80e97cd206b0e3760629299331534038a1acaaaf55159"}
Nov 03 15:23:25.028868 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.028845    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:25.191471 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.191440    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b\" (UniqueName: \"kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b\") pod \"26535a8e-c190-49a8-8818-7866725b4d78\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") "
Nov 03 15:23:25.191637 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.191490    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca\") pod \"26535a8e-c190-49a8-8818-7866725b4d78\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") "
Nov 03 15:23:25.191637 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.191539    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca\") pod \"26535a8e-c190-49a8-8818-7866725b4d78\" (UID: \"26535a8e-c190-49a8-8818-7866725b4d78\") "
Nov 03 15:23:25.191930 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.191909    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca" pod "26535a8e-c190-49a8-8818-7866725b4d78" (UID: "26535a8e-c190-49a8-8818-7866725b4d78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:23:25.193635 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.193611    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca" pod "26535a8e-c190-49a8-8818-7866725b4d78" (UID: "26535a8e-c190-49a8-8818-7866725b4d78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:23:25.193738 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.193614    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b" pod "26535a8e-c190-49a8-8818-7866725b4d78" (UID: "26535a8e-c190-49a8-8818-7866725b4d78"). InnerVolumeSpecName "kube-api-access-fn24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:23:25.292742 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.292714    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:23:25.292742 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.292739    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn24b\" (UniqueName: \"kubernetes.io/projected/26535a8e-c190-49a8-8818-7866725b4d78-kube-api-access-fn24b\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:23:25.292904 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.292749    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26535a8e-c190-49a8-8818-7866725b4d78-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:23:25.908207 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.908175    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt" event={"ID":"26535a8e-c190-49a8-8818-7866725b4d78","Type":"ContainerDied","Data":"3b06bb1d7b739819bcd12710869a46b33e30d6b6869051bf6afa3c2282a73dca"}
Nov 03 15:23:25.908207 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.908205    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d8eeeb0-pv2nt"
Nov 03 15:23:25.908526 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:25.908223    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b06bb1d7b739819bcd12710869a46b33e30d6b6869051bf6afa3c2282a73dca"
Nov 03 15:23:27.985607 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:23:27.985574    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:23:32.518287 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.518242    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"]
Nov 03 15:23:32.518770 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.518678    6481 cpu_manager.go:414] "containerMap: removing container" podUID="26535a8e-c190-49a8-8818-7866725b4d78" containerName="revoke-keys"
Nov 03 15:23:32.518770 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.518690    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26535a8e-c190-49a8-8818-7866725b4d78" containerName="revoke-keys"
Nov 03 15:23:32.518770 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.518762    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="26535a8e-c190-49a8-8818-7866725b4d78" containerName="revoke-keys"
Nov 03 15:23:32.521987 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.521967    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.537890 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.537870    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:23:32.548499 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.548475    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:23:32.550673 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.550651    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"]
Nov 03 15:23:32.609552 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.609521    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:23:32.609552 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.609521    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:23:32.660973 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.660944    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xz6m\" (UniqueName: \"kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.661114 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.660981    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.661114 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.661075    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.762656 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.762625    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.762822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.762768    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xz6m\" (UniqueName: \"kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.762822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.762807    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.763473 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.763450    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.764986 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.764968    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.783477 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.783423    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xz6m\" (UniqueName: \"kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m\") pod \"maas-api-revoke-sub-26dfdc6d-zq8d9\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:32.839282 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:32.839258    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:33.142390 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:33.142363    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"]
Nov 03 15:23:33.144750 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:23:33.144723    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6306b5f5_474d_4fa4_8b76_131f1b0ac88c.slice/crio-14fa23125d132ae38c813e4995ec5f1857cbd30f5d995db94164b99ed202e2f6 WatchSource:0}: Error finding container 14fa23125d132ae38c813e4995ec5f1857cbd30f5d995db94164b99ed202e2f6: Status 404 returned error can't find the container with id 14fa23125d132ae38c813e4995ec5f1857cbd30f5d995db94164b99ed202e2f6
Nov 03 15:23:33.477885 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:33.477860    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:23:33.952249 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:33.951273    6481 generic.go:356] "Generic (PLEG): container finished" podID="6306b5f5-474d-4fa4-8b76-131f1b0ac88c" containerID="5192b4d62efa13efdfa05eba7acaa5d0dc21f777bdce3206d32a31d2139cd36d" exitCode=0
Nov 03 15:23:33.952249 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:33.951311    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9" event={"ID":"6306b5f5-474d-4fa4-8b76-131f1b0ac88c","Type":"ContainerDied","Data":"5192b4d62efa13efdfa05eba7acaa5d0dc21f777bdce3206d32a31d2139cd36d"}
Nov 03 15:23:33.952249 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:33.951371    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9" event={"ID":"6306b5f5-474d-4fa4-8b76-131f1b0ac88c","Type":"ContainerStarted","Data":"14fa23125d132ae38c813e4995ec5f1857cbd30f5d995db94164b99ed202e2f6"}
Nov 03 15:23:35.289446 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.289098    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:35.442606 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.441576    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca\") pod \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") "
Nov 03 15:23:35.442606 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.441754    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca\") pod \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") "
Nov 03 15:23:35.442606 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.441851    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m\" (UniqueName: \"kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m\") pod \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\" (UID: \"6306b5f5-474d-4fa4-8b76-131f1b0ac88c\") "
Nov 03 15:23:35.448470 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.448442    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca" pod "6306b5f5-474d-4fa4-8b76-131f1b0ac88c" (UID: "6306b5f5-474d-4fa4-8b76-131f1b0ac88c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:23:35.474881 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.474829    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m" pod "6306b5f5-474d-4fa4-8b76-131f1b0ac88c" (UID: "6306b5f5-474d-4fa4-8b76-131f1b0ac88c"). InnerVolumeSpecName "kube-api-access-8xz6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:23:35.487431 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.484567    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca" pod "6306b5f5-474d-4fa4-8b76-131f1b0ac88c" (UID: "6306b5f5-474d-4fa4-8b76-131f1b0ac88c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:23:35.549305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.545961    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:23:35.549305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.545998    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xz6m\" (UniqueName: \"kubernetes.io/projected/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-kube-api-access-8xz6m\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:23:35.549305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.546015    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6306b5f5-474d-4fa4-8b76-131f1b0ac88c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:23:35.990253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.990117    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9" event={"ID":"6306b5f5-474d-4fa4-8b76-131f1b0ac88c","Type":"ContainerDied","Data":"14fa23125d132ae38c813e4995ec5f1857cbd30f5d995db94164b99ed202e2f6"}
Nov 03 15:23:35.990253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.990205    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14fa23125d132ae38c813e4995ec5f1857cbd30f5d995db94164b99ed202e2f6"
Nov 03 15:23:35.990557 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:35.990338    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26dfdc6d-zq8d9"
Nov 03 15:23:43.885286 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:43.885223    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:23:49.175005 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:23:49.174960    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:23:58.133514 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:23:58.133448    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:24:00.868840 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:00.868805    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:24:04.990903 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:04.990853    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:24:05.155027 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.154990    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"]
Nov 03 15:24:05.155943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.155916    6481 cpu_manager.go:414] "containerMap: removing container" podUID="6306b5f5-474d-4fa4-8b76-131f1b0ac88c" containerName="revoke-keys"
Nov 03 15:24:05.156105 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.156089    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6306b5f5-474d-4fa4-8b76-131f1b0ac88c" containerName="revoke-keys"
Nov 03 15:24:05.156409 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.156390    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="6306b5f5-474d-4fa4-8b76-131f1b0ac88c" containerName="revoke-keys"
Nov 03 15:24:05.192568 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.192524    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.235252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.215362    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"]
Nov 03 15:24:05.255297 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.243804    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tt2\" (UniqueName: \"kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.255297 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.243864    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.255297 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.243944    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.293254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.277142    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:24:05.293254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.277189    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:24:05.293254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.293015    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:24:05.302883 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.302838    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:24:05.373267 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.346109    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tt2\" (UniqueName: \"kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.373875 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.373847    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.374138 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.374116    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.376420 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.376396    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.394164 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.394104    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tt2\" (UniqueName: \"kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.400053 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.399991    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26f2e7aa-2kpjx\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.577136 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.577060    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:05.875447 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:05.875423    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"]
Nov 03 15:24:05.877395 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:24:05.877368    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3686fec0_1b49_4ca6_9df0_7b769a8ead67.slice/crio-532f7a7ab417b457e5b91a05c32f1ddb4943d800e129f484535b3dc29987c5c0 WatchSource:0}: Error finding container 532f7a7ab417b457e5b91a05c32f1ddb4943d800e129f484535b3dc29987c5c0: Status 404 returned error can't find the container with id 532f7a7ab417b457e5b91a05c32f1ddb4943d800e129f484535b3dc29987c5c0
Nov 03 15:24:06.352168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:06.352132    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx" event={"ID":"3686fec0-1b49-4ca6-9df0-7b769a8ead67","Type":"ContainerStarted","Data":"52436a11aa3903d2e9775c6de8b88699d8766ff53de319b50887d700cff878cf"}
Nov 03 15:24:06.352581 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:06.352186    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx" event={"ID":"3686fec0-1b49-4ca6-9df0-7b769a8ead67","Type":"ContainerStarted","Data":"532f7a7ab417b457e5b91a05c32f1ddb4943d800e129f484535b3dc29987c5c0"}
Nov 03 15:24:06.393372 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:06.393324    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx" podStartSLOduration=1.393312201 podStartE2EDuration="1.393312201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:06.391696659 +0000 UTC m=+4324.441727859" watchObservedRunningTime="2026-11-03 15:24:06.393312201 +0000 UTC m=+4324.443343381"
Nov 03 15:24:07.360093 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:07.360059    6481 generic.go:356] "Generic (PLEG): container finished" podID="3686fec0-1b49-4ca6-9df0-7b769a8ead67" containerID="52436a11aa3903d2e9775c6de8b88699d8766ff53de319b50887d700cff878cf" exitCode=0
Nov 03 15:24:07.360525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:07.360151    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx" event={"ID":"3686fec0-1b49-4ca6-9df0-7b769a8ead67","Type":"ContainerDied","Data":"52436a11aa3903d2e9775c6de8b88699d8766ff53de319b50887d700cff878cf"}
Nov 03 15:24:08.498970 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.498945    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:08.605497 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.605470    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2\" (UniqueName: \"kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2\") pod \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") "
Nov 03 15:24:08.605670 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.605559    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca\") pod \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") "
Nov 03 15:24:08.605670 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.605610    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca\") pod \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\" (UID: \"3686fec0-1b49-4ca6-9df0-7b769a8ead67\") "
Nov 03 15:24:08.605949 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.605927    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca" pod "3686fec0-1b49-4ca6-9df0-7b769a8ead67" (UID: "3686fec0-1b49-4ca6-9df0-7b769a8ead67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:08.607648 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.607629    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca" pod "3686fec0-1b49-4ca6-9df0-7b769a8ead67" (UID: "3686fec0-1b49-4ca6-9df0-7b769a8ead67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:24:08.607719 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.607706    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2" pod "3686fec0-1b49-4ca6-9df0-7b769a8ead67" (UID: "3686fec0-1b49-4ca6-9df0-7b769a8ead67"). InnerVolumeSpecName "kube-api-access-p4tt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:08.706751 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.706721    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4tt2\" (UniqueName: \"kubernetes.io/projected/3686fec0-1b49-4ca6-9df0-7b769a8ead67-kube-api-access-p4tt2\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:08.706751 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.706751    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:08.706950 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:08.706762    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3686fec0-1b49-4ca6-9df0-7b769a8ead67-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:09.373299 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:09.373271    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx"
Nov 03 15:24:09.373509 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:09.373260    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26f2e7aa-2kpjx" event={"ID":"3686fec0-1b49-4ca6-9df0-7b769a8ead67","Type":"ContainerDied","Data":"532f7a7ab417b457e5b91a05c32f1ddb4943d800e129f484535b3dc29987c5c0"}
Nov 03 15:24:09.373509 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:09.373391    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532f7a7ab417b457e5b91a05c32f1ddb4943d800e129f484535b3dc29987c5c0"
Nov 03 15:24:09.437914 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:09.437882    6481 scope.go:122] "RemoveContainer" containerID="97a59836b657be1d37ed5700a51819eab204ff4d1347d33473b8f65be08127b3"
Nov 03 15:24:15.644297 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.644181    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"]
Nov 03 15:24:15.644831 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.644698    6481 cpu_manager.go:414] "containerMap: removing container" podUID="3686fec0-1b49-4ca6-9df0-7b769a8ead67" containerName="revoke-keys"
Nov 03 15:24:15.644831 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.644712    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3686fec0-1b49-4ca6-9df0-7b769a8ead67" containerName="revoke-keys"
Nov 03 15:24:15.644831 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.644803    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="3686fec0-1b49-4ca6-9df0-7b769a8ead67" containerName="revoke-keys"
Nov 03 15:24:15.648085 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.648069    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.663666 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.663641    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:24:15.674747 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.674716    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:24:15.674747 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.674735    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:24:15.674979 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.674903    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:24:15.677137 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.677114    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"]
Nov 03 15:24:15.778777 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.778747    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsd6n\" (UniqueName: \"kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.778948 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.778786    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.778993 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.778944    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.879810 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.879778    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsd6n\" (UniqueName: \"kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.879973 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.879822    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.879973 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.879898    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.880547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.880524    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.882190 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.882170    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.900991 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.900928    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsd6n\" (UniqueName: \"kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n\") pod \"maas-api-revoke-sub-c3e6c43c-xg4n7\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:15.960924 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:15.960901    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:16.261106 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:16.261083    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"]
Nov 03 15:24:16.409285 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:16.409254    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7" event={"ID":"20a991b3-f88a-40be-ab50-a9d3f337d902","Type":"ContainerStarted","Data":"539a61c59c1e884eb54d19fa554dae3b6fd5f294423e8cdca78ffa448983beae"}
Nov 03 15:24:17.370293 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:17.370208    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:24:17.416139 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:17.416104    6481 generic.go:356] "Generic (PLEG): container finished" podID="20a991b3-f88a-40be-ab50-a9d3f337d902" containerID="6adae58cde69b1a6b5525cbb244620b065ffddcf17bdaf91e218f490e30f9566" exitCode=0
Nov 03 15:24:17.416343 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:17.416187    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7" event={"ID":"20a991b3-f88a-40be-ab50-a9d3f337d902","Type":"ContainerDied","Data":"6adae58cde69b1a6b5525cbb244620b065ffddcf17bdaf91e218f490e30f9566"}
Nov 03 15:24:18.610304 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.610271    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:18.710988 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.710891    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca\") pod \"20a991b3-f88a-40be-ab50-a9d3f337d902\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") "
Nov 03 15:24:18.710988 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.710973    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca\") pod \"20a991b3-f88a-40be-ab50-a9d3f337d902\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") "
Nov 03 15:24:18.711260 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.711029    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n\" (UniqueName: \"kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n\") pod \"20a991b3-f88a-40be-ab50-a9d3f337d902\" (UID: \"20a991b3-f88a-40be-ab50-a9d3f337d902\") "
Nov 03 15:24:18.711901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.711770    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca" pod "20a991b3-f88a-40be-ab50-a9d3f337d902" (UID: "20a991b3-f88a-40be-ab50-a9d3f337d902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:18.713982 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.713938    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca" pod "20a991b3-f88a-40be-ab50-a9d3f337d902" (UID: "20a991b3-f88a-40be-ab50-a9d3f337d902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:24:18.713982 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.713941    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n" pod "20a991b3-f88a-40be-ab50-a9d3f337d902" (UID: "20a991b3-f88a-40be-ab50-a9d3f337d902"). InnerVolumeSpecName "kube-api-access-qsd6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:18.812334 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.812285    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:18.812334 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.812329    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20a991b3-f88a-40be-ab50-a9d3f337d902-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:18.812621 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:18.812345    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsd6n\" (UniqueName: \"kubernetes.io/projected/20a991b3-f88a-40be-ab50-a9d3f337d902-kube-api-access-qsd6n\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:19.447034 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:19.446996    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7" event={"ID":"20a991b3-f88a-40be-ab50-a9d3f337d902","Type":"ContainerDied","Data":"539a61c59c1e884eb54d19fa554dae3b6fd5f294423e8cdca78ffa448983beae"}
Nov 03 15:24:19.451367 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:19.451345    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539a61c59c1e884eb54d19fa554dae3b6fd5f294423e8cdca78ffa448983beae"
Nov 03 15:24:19.451570 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:19.451555    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3e6c43c-xg4n7"
Nov 03 15:24:26.018986 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.018947    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"]
Nov 03 15:24:26.019446 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.019432    6481 cpu_manager.go:414] "containerMap: removing container" podUID="20a991b3-f88a-40be-ab50-a9d3f337d902" containerName="revoke-keys"
Nov 03 15:24:26.019503 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.019448    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a991b3-f88a-40be-ab50-a9d3f337d902" containerName="revoke-keys"
Nov 03 15:24:26.019543 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.019515    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a991b3-f88a-40be-ab50-a9d3f337d902" containerName="revoke-keys"
Nov 03 15:24:26.022846 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.022827    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.050260 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.049405    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:24:26.050260 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.049489    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:24:26.050260 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.049494    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:24:26.050260 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.049579    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:24:26.052300 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.052274    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"]
Nov 03 15:24:26.086632 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.086599    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.086788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.086734    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtzn\" (UniqueName: \"kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.086788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.086765    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.187210 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.187173    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.187426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.187313    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtzn\" (UniqueName: \"kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.187426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.187352    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.187945 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.187920    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.189447 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.189427    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.208038 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.208005    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtzn\" (UniqueName: \"kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn\") pod \"maas-api-revoke-sub-3332baab-ngjhm\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.336060 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.335984    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:26.651565 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:26.633965    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"]
Nov 03 15:24:26.651565 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:24:26.635265    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77e2c37_353f_40b8_a7f2_48f51d5e151c.slice/crio-8908efbb369e4d1f070c17fb3336f0b9be828356d54c0e9c4454a368300319b7 WatchSource:0}: Error finding container 8908efbb369e4d1f070c17fb3336f0b9be828356d54c0e9c4454a368300319b7: Status 404 returned error can't find the container with id 8908efbb369e4d1f070c17fb3336f0b9be828356d54c0e9c4454a368300319b7
Nov 03 15:24:27.213005 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:27.212971    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:24:27.519350 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:27.519270    6481 generic.go:356] "Generic (PLEG): container finished" podID="e77e2c37-353f-40b8-a7f2-48f51d5e151c" containerID="8baeef2c6c3194bebe7f00ed58933149b2f2a252d3cce066aa171baab2244679" exitCode=0
Nov 03 15:24:27.519502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:27.519362    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm" event={"ID":"e77e2c37-353f-40b8-a7f2-48f51d5e151c","Type":"ContainerDied","Data":"8baeef2c6c3194bebe7f00ed58933149b2f2a252d3cce066aa171baab2244679"}
Nov 03 15:24:27.519502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:27.519414    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm" event={"ID":"e77e2c37-353f-40b8-a7f2-48f51d5e151c","Type":"ContainerStarted","Data":"8908efbb369e4d1f070c17fb3336f0b9be828356d54c0e9c4454a368300319b7"}
Nov 03 15:24:27.985896 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:24:27.985862    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:24:28.658120 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.658092    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:28.811806 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.811716    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn\" (UniqueName: \"kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn\") pod \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") "
Nov 03 15:24:28.811961 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.811849    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca\") pod \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") "
Nov 03 15:24:28.811961 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.811870    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca\") pod \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\" (UID: \"e77e2c37-353f-40b8-a7f2-48f51d5e151c\") "
Nov 03 15:24:28.812203 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.812176    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca" pod "e77e2c37-353f-40b8-a7f2-48f51d5e151c" (UID: "e77e2c37-353f-40b8-a7f2-48f51d5e151c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:28.813801 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.813779    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca" pod "e77e2c37-353f-40b8-a7f2-48f51d5e151c" (UID: "e77e2c37-353f-40b8-a7f2-48f51d5e151c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:24:28.813959 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.813941    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn" pod "e77e2c37-353f-40b8-a7f2-48f51d5e151c" (UID: "e77e2c37-353f-40b8-a7f2-48f51d5e151c"). InnerVolumeSpecName "kube-api-access-bmtzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:28.913022 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.912991    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:28.913022 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.913018    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77e2c37-353f-40b8-a7f2-48f51d5e151c-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:28.913022 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.913027    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmtzn\" (UniqueName: \"kubernetes.io/projected/e77e2c37-353f-40b8-a7f2-48f51d5e151c-kube-api-access-bmtzn\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:28.993215 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:28.993187    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:24:29.531401 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:29.531375    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm"
Nov 03 15:24:29.531607 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:29.531368    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3332baab-ngjhm" event={"ID":"e77e2c37-353f-40b8-a7f2-48f51d5e151c","Type":"ContainerDied","Data":"8908efbb369e4d1f070c17fb3336f0b9be828356d54c0e9c4454a368300319b7"}
Nov 03 15:24:29.531607 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:29.531503    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8908efbb369e4d1f070c17fb3336f0b9be828356d54c0e9c4454a368300319b7"
Nov 03 15:24:36.412735 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.412701    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"]
Nov 03 15:24:36.413152 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.413116    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e77e2c37-353f-40b8-a7f2-48f51d5e151c" containerName="revoke-keys"
Nov 03 15:24:36.413152 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.413127    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77e2c37-353f-40b8-a7f2-48f51d5e151c" containerName="revoke-keys"
Nov 03 15:24:36.413262 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.413198    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77e2c37-353f-40b8-a7f2-48f51d5e151c" containerName="revoke-keys"
Nov 03 15:24:36.416548 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.416526    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.443416 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.443379    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:24:36.443581 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.443438    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:24:36.443581 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.443523    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:24:36.443732 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.443601    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:24:36.446923 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.446898    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"]
Nov 03 15:24:36.590162 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.590124    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.590162 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.590164    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8bh\" (UniqueName: \"kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.590412 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.590322    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.691298 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.691196    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.691298 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.691251    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8bh\" (UniqueName: \"kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.691545 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.691328    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.691838 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.691817    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.693684 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.693657    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.712146 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.712123    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8bh\" (UniqueName: \"kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh\") pod \"maas-api-revoke-sub-922c97e6-tlkc2\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:36.729800 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:36.729781    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:37.030199 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:37.030176    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"]
Nov 03 15:24:37.031856 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:24:37.031827    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc6c93b_a854_41c5_a67a_9a0f878e2bc3.slice/crio-750ebc7fccf1ab7ca867e69f9a37cdf13f333844c3bad135d5453c2aae2b4232 WatchSource:0}: Error finding container 750ebc7fccf1ab7ca867e69f9a37cdf13f333844c3bad135d5453c2aae2b4232: Status 404 returned error can't find the container with id 750ebc7fccf1ab7ca867e69f9a37cdf13f333844c3bad135d5453c2aae2b4232
Nov 03 15:24:37.578202 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:37.578164    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2" event={"ID":"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3","Type":"ContainerStarted","Data":"9307ab3f4ec8b24c56a9fc6a5a7454c9b19e3095f5316eabc20645f2ac238be9"}
Nov 03 15:24:37.578626 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:37.578239    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2" event={"ID":"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3","Type":"ContainerStarted","Data":"750ebc7fccf1ab7ca867e69f9a37cdf13f333844c3bad135d5453c2aae2b4232"}
Nov 03 15:24:37.621384 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:37.621337    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2" podStartSLOduration=1.621322395 podStartE2EDuration="1.621322395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:24:37.619245897 +0000 UTC m=+4355.669277074" watchObservedRunningTime="2026-11-03 15:24:37.621322395 +0000 UTC m=+4355.671353577"
Nov 03 15:24:38.590297 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:38.590215    6481 generic.go:356] "Generic (PLEG): container finished" podID="2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" containerID="9307ab3f4ec8b24c56a9fc6a5a7454c9b19e3095f5316eabc20645f2ac238be9" exitCode=0
Nov 03 15:24:38.590836 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:38.590309    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2" event={"ID":"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3","Type":"ContainerDied","Data":"9307ab3f4ec8b24c56a9fc6a5a7454c9b19e3095f5316eabc20645f2ac238be9"}
Nov 03 15:24:40.348777 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.348308    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:24:40.348777 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.348372    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-z6fd6" podUID="7285fd08-91c8-4113-9f66-d5e6810b02d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.20:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:24:40.412917 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.412403    6481 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 15:24:40.412917 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.412471    6481 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-2p5f4" podUID="0b5ad832-aa94-4941-9b2e-852a9ddfda03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 15:24:40.481252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.473392    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:40.609262 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.599404    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca\") pod \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") "
Nov 03 15:24:40.609262 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.599524    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca\") pod \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") "
Nov 03 15:24:40.609262 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.599594    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh\" (UniqueName: \"kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh\") pod \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\" (UID: \"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3\") "
Nov 03 15:24:40.620454 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.620419    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca" pod "2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" (UID: "2fc6c93b-a854-41c5-a67a-9a0f878e2bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:40.698661 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.698290    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2" event={"ID":"2fc6c93b-a854-41c5-a67a-9a0f878e2bc3","Type":"ContainerDied","Data":"750ebc7fccf1ab7ca867e69f9a37cdf13f333844c3bad135d5453c2aae2b4232"}
Nov 03 15:24:40.698661 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.698358    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750ebc7fccf1ab7ca867e69f9a37cdf13f333844c3bad135d5453c2aae2b4232"
Nov 03 15:24:40.698661 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.698431    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-922c97e6-tlkc2"
Nov 03 15:24:40.700808 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.700776    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:40.701541 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.701515    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh" pod "2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" (UID: "2fc6c93b-a854-41c5-a67a-9a0f878e2bc3"). InnerVolumeSpecName "kube-api-access-jt8bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:40.720066 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.720033    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca" pod "2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" (UID: "2fc6c93b-a854-41c5-a67a-9a0f878e2bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:24:40.802399 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.802367    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:40.802665 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:40.802646    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8bh\" (UniqueName: \"kubernetes.io/projected/2fc6c93b-a854-41c5-a67a-9a0f878e2bc3-kube-api-access-jt8bh\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:46.855329 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.855291    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"]
Nov 03 15:24:46.855924 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.855849    6481 cpu_manager.go:414] "containerMap: removing container" podUID="2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" containerName="revoke-keys"
Nov 03 15:24:46.855924 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.855869    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" containerName="revoke-keys"
Nov 03 15:24:46.856025 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.855967    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc6c93b-a854-41c5-a67a-9a0f878e2bc3" containerName="revoke-keys"
Nov 03 15:24:46.859270 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.859249    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.877351 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.877308    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.877515 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.877405    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.877515 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.877437    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfln\" (UniqueName: \"kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.880713 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.880688    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"]
Nov 03 15:24:46.888277 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.888256    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:24:46.888496 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.888473    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:24:46.888621 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.888534    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:24:46.888621 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.888606    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:24:46.978596 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.978567    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.978596 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.978597    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfln\" (UniqueName: \"kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.978797 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.978680    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.979236 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.979207    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.980864 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.980842    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:46.999052 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:46.999031    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfln\" (UniqueName: \"kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln\") pod \"maas-api-revoke-sub-0013a6d7-z6fkg\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:47.172729 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:47.172703    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:47.475288 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:47.475251    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"]
Nov 03 15:24:47.477031 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:24:47.477006    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d47603_32f0_4ff8_ae44_eafa13ee3c90.slice/crio-d5ae17d797441c4224573d0dfda68f440de064e97a013990a06ae5cabfeb8c31 WatchSource:0}: Error finding container d5ae17d797441c4224573d0dfda68f440de064e97a013990a06ae5cabfeb8c31: Status 404 returned error can't find the container with id d5ae17d797441c4224573d0dfda68f440de064e97a013990a06ae5cabfeb8c31
Nov 03 15:24:47.766310 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:47.766236    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg" event={"ID":"87d47603-32f0-4ff8-ae44-eafa13ee3c90","Type":"ContainerStarted","Data":"d5ae17d797441c4224573d0dfda68f440de064e97a013990a06ae5cabfeb8c31"}
Nov 03 15:24:48.779194 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:48.779147    6481 generic.go:356] "Generic (PLEG): container finished" podID="87d47603-32f0-4ff8-ae44-eafa13ee3c90" containerID="fc836a4ec95d7833ff780011422146c0cc89b9d8fda40778cc11e00228b0350a" exitCode=0
Nov 03 15:24:48.779727 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:48.779369    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg" event={"ID":"87d47603-32f0-4ff8-ae44-eafa13ee3c90","Type":"ContainerDied","Data":"fc836a4ec95d7833ff780011422146c0cc89b9d8fda40778cc11e00228b0350a"}
Nov 03 15:24:50.513449 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.513416    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:50.665304 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.661424    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln\" (UniqueName: \"kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln\") pod \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") "
Nov 03 15:24:50.665304 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.661502    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca\") pod \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") "
Nov 03 15:24:50.665304 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.661598    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca\") pod \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\" (UID: \"87d47603-32f0-4ff8-ae44-eafa13ee3c90\") "
Nov 03 15:24:50.674250 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.667183    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca" pod "87d47603-32f0-4ff8-ae44-eafa13ee3c90" (UID: "87d47603-32f0-4ff8-ae44-eafa13ee3c90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:24:50.696032 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.688369    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca" pod "87d47603-32f0-4ff8-ae44-eafa13ee3c90" (UID: "87d47603-32f0-4ff8-ae44-eafa13ee3c90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:24:50.696032 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.695754    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln" pod "87d47603-32f0-4ff8-ae44-eafa13ee3c90" (UID: "87d47603-32f0-4ff8-ae44-eafa13ee3c90"). InnerVolumeSpecName "kube-api-access-4cfln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:24:50.763014 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.762916    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:50.763014 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.762970    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfln\" (UniqueName: \"kubernetes.io/projected/87d47603-32f0-4ff8-ae44-eafa13ee3c90-kube-api-access-4cfln\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:50.763014 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.762986    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87d47603-32f0-4ff8-ae44-eafa13ee3c90-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:24:50.835770 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.835722    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg" event={"ID":"87d47603-32f0-4ff8-ae44-eafa13ee3c90","Type":"ContainerDied","Data":"d5ae17d797441c4224573d0dfda68f440de064e97a013990a06ae5cabfeb8c31"}
Nov 03 15:24:50.836092 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.836053    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ae17d797441c4224573d0dfda68f440de064e97a013990a06ae5cabfeb8c31"
Nov 03 15:24:50.836444 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:50.836320    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0013a6d7-z6fkg"
Nov 03 15:24:57.221169 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.221134    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"]
Nov 03 15:24:57.222052 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.222031    6481 cpu_manager.go:414] "containerMap: removing container" podUID="87d47603-32f0-4ff8-ae44-eafa13ee3c90" containerName="revoke-keys"
Nov 03 15:24:57.222052 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.222053    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d47603-32f0-4ff8-ae44-eafa13ee3c90" containerName="revoke-keys"
Nov 03 15:24:57.222168 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.222119    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d47603-32f0-4ff8-ae44-eafa13ee3c90" containerName="revoke-keys"
Nov 03 15:24:57.225542 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.225521    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.247311 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.247282    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"]
Nov 03 15:24:57.258724 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.257593    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:24:57.258724 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.257728    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:24:57.258724 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.257797    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:24:57.258724 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.258126    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:24:57.352613 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.352580    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rrrh\" (UniqueName: \"kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.352811 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.352621    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.352811 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.352645    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.453782 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.453736    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rrrh\" (UniqueName: \"kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.453782 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.453783    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.454052 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.453803    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.454549 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.454525    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.456283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.456248    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.474156 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.474090    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rrrh\" (UniqueName: \"kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh\") pod \"maas-api-revoke-sub-1145cec5-dlzk2\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.548361 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.548320    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:24:57.859070 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.859045    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"]
Nov 03 15:24:57.860939 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:24:57.860908    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c3b90a_a636_4422_9bb5_2ed66b6896f0.slice/crio-f97e383121a24f1bc38fe492c827c90053a5b7a00c50242e8a956e183c40db8a WatchSource:0}: Error finding container f97e383121a24f1bc38fe492c827c90053a5b7a00c50242e8a956e183c40db8a: Status 404 returned error can't find the container with id f97e383121a24f1bc38fe492c827c90053a5b7a00c50242e8a956e183c40db8a
Nov 03 15:24:57.880603 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:57.880573    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2" event={"ID":"f5c3b90a-a636-4422-9bb5-2ed66b6896f0","Type":"ContainerStarted","Data":"f97e383121a24f1bc38fe492c827c90053a5b7a00c50242e8a956e183c40db8a"}
Nov 03 15:24:57.985931 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:24:57.985896    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:24:58.890893 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:58.890792    6481 generic.go:356] "Generic (PLEG): container finished" podID="f5c3b90a-a636-4422-9bb5-2ed66b6896f0" containerID="01a94ca19ebfa19cbe79aa1ae1a05e23d590108bfe91e68fe01fdaa3269752d5" exitCode=0
Nov 03 15:24:58.891413 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:24:58.890877    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2" event={"ID":"f5c3b90a-a636-4422-9bb5-2ed66b6896f0","Type":"ContainerDied","Data":"01a94ca19ebfa19cbe79aa1ae1a05e23d590108bfe91e68fe01fdaa3269752d5"}
Nov 03 15:25:00.631252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.630406    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:25:00.696658 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.696626    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca\") pod \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") "
Nov 03 15:25:00.697309 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.697286    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca\") pod \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") "
Nov 03 15:25:00.698017 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.697995    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh\" (UniqueName: \"kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh\") pod \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\" (UID: \"f5c3b90a-a636-4422-9bb5-2ed66b6896f0\") "
Nov 03 15:25:00.698895 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.697205    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca" pod "f5c3b90a-a636-4422-9bb5-2ed66b6896f0" (UID: "f5c3b90a-a636-4422-9bb5-2ed66b6896f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:25:00.731396 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.728802    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca" pod "f5c3b90a-a636-4422-9bb5-2ed66b6896f0" (UID: "f5c3b90a-a636-4422-9bb5-2ed66b6896f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:25:00.731396 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.731356    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh" pod "f5c3b90a-a636-4422-9bb5-2ed66b6896f0" (UID: "f5c3b90a-a636-4422-9bb5-2ed66b6896f0"). InnerVolumeSpecName "kube-api-access-8rrrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:25:00.809305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.803024    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rrrh\" (UniqueName: \"kubernetes.io/projected/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-kube-api-access-8rrrh\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:00.809305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.803064    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:00.809305 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.803081    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c3b90a-a636-4422-9bb5-2ed66b6896f0-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:00.931092 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.931054    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2" event={"ID":"f5c3b90a-a636-4422-9bb5-2ed66b6896f0","Type":"ContainerDied","Data":"f97e383121a24f1bc38fe492c827c90053a5b7a00c50242e8a956e183c40db8a"}
Nov 03 15:25:00.931390 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.931367    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f97e383121a24f1bc38fe492c827c90053a5b7a00c50242e8a956e183c40db8a"
Nov 03 15:25:00.931561 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:00.931424    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1145cec5-dlzk2"
Nov 03 15:25:05.136736 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:05.136704    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:25:07.731097 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.731066    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"]
Nov 03 15:25:07.731518 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.731500    6481 cpu_manager.go:414] "containerMap: removing container" podUID="f5c3b90a-a636-4422-9bb5-2ed66b6896f0" containerName="revoke-keys"
Nov 03 15:25:07.731518 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.731511    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c3b90a-a636-4422-9bb5-2ed66b6896f0" containerName="revoke-keys"
Nov 03 15:25:07.731657 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.731591    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c3b90a-a636-4422-9bb5-2ed66b6896f0" containerName="revoke-keys"
Nov 03 15:25:07.734666 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.734647    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.763767 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.763722    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:25:07.763767 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.763740    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:25:07.763975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.763785    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:25:07.763975 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.763885    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:25:07.767717 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.767693    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"]
Nov 03 15:25:07.815374 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.815337    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76k8\" (UniqueName: \"kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.815520 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.815461    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.815570 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.815523    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.916721 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.916691    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.916863 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.916750    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76k8\" (UniqueName: \"kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.916863 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.916804    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.917485 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.917463    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.919129 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.919102    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:07.938488 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:07.938461    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76k8\" (UniqueName: \"kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8\") pod \"maas-api-revoke-sub-48ed8ae3-9db2c\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:08.047171 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:08.047102    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:08.355071 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:08.355030    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"]
Nov 03 15:25:08.992187 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:08.992151    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c" event={"ID":"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e","Type":"ContainerStarted","Data":"304495370def38c12069ce8cd03bb84b9f4e8e6730aa534733c8b54d5e14d88f"}
Nov 03 15:25:08.992599 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:08.992209    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c" event={"ID":"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e","Type":"ContainerStarted","Data":"d621672697e1273bc960c648215f2f177ee0be065c0bdc55562c418b6ace9445"}
Nov 03 15:25:09.034092 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:09.034034    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c" podStartSLOduration=2.034018936 podStartE2EDuration="2.034018936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:09.033689965 +0000 UTC m=+4387.083721160" watchObservedRunningTime="2026-11-03 15:25:09.034018936 +0000 UTC m=+4387.084050119"
Nov 03 15:25:10.002603 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:10.002564    6481 generic.go:356] "Generic (PLEG): container finished" podID="9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" containerID="304495370def38c12069ce8cd03bb84b9f4e8e6730aa534733c8b54d5e14d88f" exitCode=0
Nov 03 15:25:10.003136 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:10.002622    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c" event={"ID":"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e","Type":"ContainerDied","Data":"304495370def38c12069ce8cd03bb84b9f4e8e6730aa534733c8b54d5e14d88f"}
Nov 03 15:25:11.623330 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.623270    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:11.750822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.750778    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca\") pod \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") "
Nov 03 15:25:11.751547 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.751523    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca\") pod \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") "
Nov 03 15:25:11.770602 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.770474    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca" pod "9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" (UID: "9024c37e-41aa-4f69-aa50-f6d31bc7ca8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:25:11.770957 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.770937    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8\" (UniqueName: \"kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8\") pod \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\" (UID: \"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e\") "
Nov 03 15:25:11.796747 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.796713    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:11.836802 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.836700    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8" pod "9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" (UID: "9024c37e-41aa-4f69-aa50-f6d31bc7ca8e"). InnerVolumeSpecName "kube-api-access-m76k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:25:11.843427 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.843379    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca" pod "9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" (UID: "9024c37e-41aa-4f69-aa50-f6d31bc7ca8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:25:11.899138 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.899031    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m76k8\" (UniqueName: \"kubernetes.io/projected/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-kube-api-access-m76k8\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:11.899138 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:11.899071    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9024c37e-41aa-4f69-aa50-f6d31bc7ca8e-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:12.043265 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:12.042406    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c"
Nov 03 15:25:12.055787 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:12.042312    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48ed8ae3-9db2c" event={"ID":"9024c37e-41aa-4f69-aa50-f6d31bc7ca8e","Type":"ContainerDied","Data":"d621672697e1273bc960c648215f2f177ee0be065c0bdc55562c418b6ace9445"}
Nov 03 15:25:12.055787 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:12.055494    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d621672697e1273bc960c648215f2f177ee0be065c0bdc55562c418b6ace9445"
Nov 03 15:25:18.093939 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.093902    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"]
Nov 03 15:25:18.094586 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.094544    6481 cpu_manager.go:414] "containerMap: removing container" podUID="9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" containerName="revoke-keys"
Nov 03 15:25:18.094586 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.094565    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" containerName="revoke-keys"
Nov 03 15:25:18.094717 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.094706    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="9024c37e-41aa-4f69-aa50-f6d31bc7ca8e" containerName="revoke-keys"
Nov 03 15:25:18.098111 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.098082    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.120490 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.120465    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"]
Nov 03 15:25:18.128641 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.128312    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:25:18.128641 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.128519    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:25:18.128641 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.128612    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:25:18.128915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.128681    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:25:18.190077 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.190044    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.190255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.190185    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sss8q\" (UniqueName: \"kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.190389 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.190370    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.291595 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.291562    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.291767 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.291628    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sss8q\" (UniqueName: \"kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.291767 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.291712    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.292360 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.292341    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.293798 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.293779    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.313328 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.313305    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sss8q\" (UniqueName: \"kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q\") pod \"maas-api-revoke-sub-2930fb36-wmwqb\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.412572 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.412543    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:18.713503 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:18.713479    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"]
Nov 03 15:25:18.715514 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:25:18.715489    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462cc4fc_b9a3_4064_950d_e76a79333ba6.slice/crio-7239bba491fb6ad9565f77d02b5665b2ac7cc99d9fd3ef0b70fcb6339f42b780 WatchSource:0}: Error finding container 7239bba491fb6ad9565f77d02b5665b2ac7cc99d9fd3ef0b70fcb6339f42b780: Status 404 returned error can't find the container with id 7239bba491fb6ad9565f77d02b5665b2ac7cc99d9fd3ef0b70fcb6339f42b780
Nov 03 15:25:19.085873 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:19.085839    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb" event={"ID":"462cc4fc-b9a3-4064-950d-e76a79333ba6","Type":"ContainerStarted","Data":"f61d761eefa4fcbc8d64cdcc9667cc1f66ad6b3dbcb965f25ca6b34835e616c8"}
Nov 03 15:25:19.086042 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:19.085898    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb" event={"ID":"462cc4fc-b9a3-4064-950d-e76a79333ba6","Type":"ContainerStarted","Data":"7239bba491fb6ad9565f77d02b5665b2ac7cc99d9fd3ef0b70fcb6339f42b780"}
Nov 03 15:25:19.125265 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:19.125198    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb" podStartSLOduration=1.1251864089999999 podStartE2EDuration="1.125186409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:19.12497957 +0000 UTC m=+4397.175010752" watchObservedRunningTime="2026-11-03 15:25:19.125186409 +0000 UTC m=+4397.175217583"
Nov 03 15:25:20.095903 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:20.095864    6481 generic.go:356] "Generic (PLEG): container finished" podID="462cc4fc-b9a3-4064-950d-e76a79333ba6" containerID="f61d761eefa4fcbc8d64cdcc9667cc1f66ad6b3dbcb965f25ca6b34835e616c8" exitCode=0
Nov 03 15:25:20.095903 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:20.095911    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb" event={"ID":"462cc4fc-b9a3-4064-950d-e76a79333ba6","Type":"ContainerDied","Data":"f61d761eefa4fcbc8d64cdcc9667cc1f66ad6b3dbcb965f25ca6b34835e616c8"}
Nov 03 15:25:21.783351 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.783320    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:21.878941 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.878840    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q\" (UniqueName: \"kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q\") pod \"462cc4fc-b9a3-4064-950d-e76a79333ba6\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") "
Nov 03 15:25:21.889399 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.889371    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca\") pod \"462cc4fc-b9a3-4064-950d-e76a79333ba6\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") "
Nov 03 15:25:21.912370 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.912342    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca\") pod \"462cc4fc-b9a3-4064-950d-e76a79333ba6\" (UID: \"462cc4fc-b9a3-4064-950d-e76a79333ba6\") "
Nov 03 15:25:21.921454 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.921427    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca" pod "462cc4fc-b9a3-4064-950d-e76a79333ba6" (UID: "462cc4fc-b9a3-4064-950d-e76a79333ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:25:21.930667 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.930641    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q" pod "462cc4fc-b9a3-4064-950d-e76a79333ba6" (UID: "462cc4fc-b9a3-4064-950d-e76a79333ba6"). InnerVolumeSpecName "kube-api-access-sss8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:25:21.930910 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:21.930888    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca" pod "462cc4fc-b9a3-4064-950d-e76a79333ba6" (UID: "462cc4fc-b9a3-4064-950d-e76a79333ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:25:22.024274 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:22.014939    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:22.024274 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:22.014978    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sss8q\" (UniqueName: \"kubernetes.io/projected/462cc4fc-b9a3-4064-950d-e76a79333ba6-kube-api-access-sss8q\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:22.024274 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:22.014993    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/462cc4fc-b9a3-4064-950d-e76a79333ba6-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:22.192047 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:22.192006    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb" event={"ID":"462cc4fc-b9a3-4064-950d-e76a79333ba6","Type":"ContainerDied","Data":"7239bba491fb6ad9565f77d02b5665b2ac7cc99d9fd3ef0b70fcb6339f42b780"}
Nov 03 15:25:22.205264 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:22.192337    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2930fb36-wmwqb"
Nov 03 15:25:22.249907 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:22.249876    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7239bba491fb6ad9565f77d02b5665b2ac7cc99d9fd3ef0b70fcb6339f42b780"
Nov 03 15:25:27.986414 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:25:27.986378    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:25:28.551176 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.551143    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"]
Nov 03 15:25:28.551690 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.551669    6481 cpu_manager.go:414] "containerMap: removing container" podUID="462cc4fc-b9a3-4064-950d-e76a79333ba6" containerName="revoke-keys"
Nov 03 15:25:28.551794 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.551692    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462cc4fc-b9a3-4064-950d-e76a79333ba6" containerName="revoke-keys"
Nov 03 15:25:28.551862 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.551814    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="462cc4fc-b9a3-4064-950d-e76a79333ba6" containerName="revoke-keys"
Nov 03 15:25:28.555206 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.555186    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.576813 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.576786    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"]
Nov 03 15:25:28.586105 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.586070    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:25:28.586105 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.586104    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:25:28.586339 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.586084    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:25:28.586339 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.586159    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:25:28.635358 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.635328    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d89r\" (UniqueName: \"kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.635499 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.635362    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.635553 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.635520    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.736357 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.736327    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d89r\" (UniqueName: \"kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.736525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.736361    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.736525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.736440    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.737136 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.737114    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.738679 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.738661    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.757649 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.757626    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d89r\" (UniqueName: \"kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r\") pod \"maas-api-revoke-sub-a11c6fa9-z2gdf\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:28.868764 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:28.868692    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:29.170524 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:29.170490    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"]
Nov 03 15:25:29.171772 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:25:29.171745    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea6a2af_3bf7_47c9_8d9a_10b1769453f5.slice/crio-3c2659fd937d62263ca18dcbc9d923bd03cc5b893688c7e2ec3d1c0a690b8c08 WatchSource:0}: Error finding container 3c2659fd937d62263ca18dcbc9d923bd03cc5b893688c7e2ec3d1c0a690b8c08: Status 404 returned error can't find the container with id 3c2659fd937d62263ca18dcbc9d923bd03cc5b893688c7e2ec3d1c0a690b8c08
Nov 03 15:25:29.229915 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:29.229880    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf" event={"ID":"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5","Type":"ContainerStarted","Data":"3c2659fd937d62263ca18dcbc9d923bd03cc5b893688c7e2ec3d1c0a690b8c08"}
Nov 03 15:25:30.242810 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:30.242764    6481 generic.go:356] "Generic (PLEG): container finished" podID="8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" containerID="64533a7d3ca35b47a678a9310cd9b29a5846ca9bf11cc85d1c74494261ae9e78" exitCode=0
Nov 03 15:25:30.243342 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:30.242862    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf" event={"ID":"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5","Type":"ContainerDied","Data":"64533a7d3ca35b47a678a9310cd9b29a5846ca9bf11cc85d1c74494261ae9e78"}
Nov 03 15:25:32.044712 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.044386    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:32.275983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.274784    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r\" (UniqueName: \"kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r\") pod \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") "
Nov 03 15:25:32.275983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.274858    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca\") pod \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") "
Nov 03 15:25:32.275983 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.274966    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca\") pod \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\" (UID: \"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5\") "
Nov 03 15:25:32.336680 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.336642    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca" pod "8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" (UID: "8ea6a2af-3bf7-47c9-8d9a-10b1769453f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:25:32.367788 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.367749    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r" pod "8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" (UID: "8ea6a2af-3bf7-47c9-8d9a-10b1769453f5"). InnerVolumeSpecName "kube-api-access-8d89r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:25:32.376377 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.369679    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca" pod "8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" (UID: "8ea6a2af-3bf7-47c9-8d9a-10b1769453f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:25:32.395257 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.377873    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d89r\" (UniqueName: \"kubernetes.io/projected/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-kube-api-access-8d89r\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:32.395530 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.395492    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:32.395688 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.395671    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea6a2af-3bf7-47c9-8d9a-10b1769453f5-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:32.395832 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.379715    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf" event={"ID":"8ea6a2af-3bf7-47c9-8d9a-10b1769453f5","Type":"ContainerDied","Data":"3c2659fd937d62263ca18dcbc9d923bd03cc5b893688c7e2ec3d1c0a690b8c08"}
Nov 03 15:25:32.414252 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.379819    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a11c6fa9-z2gdf"
Nov 03 15:25:32.419189 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:32.418370    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c2659fd937d62263ca18dcbc9d923bd03cc5b893688c7e2ec3d1c0a690b8c08"
Nov 03 15:25:38.926434 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.926400    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"]
Nov 03 15:25:38.927066 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.927045    6481 cpu_manager.go:414] "containerMap: removing container" podUID="8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" containerName="revoke-keys"
Nov 03 15:25:38.927205 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.927067    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" containerName="revoke-keys"
Nov 03 15:25:38.927303 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.927250    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea6a2af-3bf7-47c9-8d9a-10b1769453f5" containerName="revoke-keys"
Nov 03 15:25:38.930461 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.930441    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:38.958093 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.958065    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:25:38.958220 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.958113    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:25:38.962488 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.962461    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"]
Nov 03 15:25:38.974143 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.974115    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:25:38.974268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.974140    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:25:38.987822 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.987791    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:38.987965 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.987841    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:38.987965 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:38.987917    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqlc\" (UniqueName: \"kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.089300 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.089267    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.089461 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.089309    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.089461 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.089407    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqlc\" (UniqueName: \"kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.089948 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.089926    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.091748 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.091720    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.111663 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.111642    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqlc\" (UniqueName: \"kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc\") pod \"maas-api-revoke-sub-c681e757-vrp5x\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.243388 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.243327    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:39.541261 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:39.541214    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"]
Nov 03 15:25:39.542267 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:25:39.542220    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fa10d6_c11c_4d49_b2bc_7bdb9e633bd7.slice/crio-8b3f5ca2ea6b4a02a0590ab6c2a414a723bc35c045bf1affe65c9956e66bb9e1 WatchSource:0}: Error finding container 8b3f5ca2ea6b4a02a0590ab6c2a414a723bc35c045bf1affe65c9956e66bb9e1: Status 404 returned error can't find the container with id 8b3f5ca2ea6b4a02a0590ab6c2a414a723bc35c045bf1affe65c9956e66bb9e1
Nov 03 15:25:40.438730 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:40.438487    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x" event={"ID":"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7","Type":"ContainerStarted","Data":"b83f9ee35c87b709a2ee6295eb005db097b4ef856e1662234e37e10ca500fadf"}
Nov 03 15:25:40.439156 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:40.438775    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x" event={"ID":"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7","Type":"ContainerStarted","Data":"8b3f5ca2ea6b4a02a0590ab6c2a414a723bc35c045bf1affe65c9956e66bb9e1"}
Nov 03 15:25:41.463108 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:41.463073    6481 generic.go:356] "Generic (PLEG): container finished" podID="22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" containerID="b83f9ee35c87b709a2ee6295eb005db097b4ef856e1662234e37e10ca500fadf" exitCode=0
Nov 03 15:25:41.463895 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:41.463867    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x" event={"ID":"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7","Type":"ContainerDied","Data":"b83f9ee35c87b709a2ee6295eb005db097b4ef856e1662234e37e10ca500fadf"}
Nov 03 15:25:41.641958 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:41.641852    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:25:41.882690 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:41.880327    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:25:42.111564 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.111537    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:42.145121 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.145077    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca\") pod \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") "
Nov 03 15:25:42.145731 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.145708    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc\" (UniqueName: \"kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc\") pod \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") "
Nov 03 15:25:42.146125 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.146096    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca\") pod \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\" (UID: \"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7\") "
Nov 03 15:25:42.151530 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.150426    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca" pod "22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" (UID: "22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:25:42.164293 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.162381    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc" pod "22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" (UID: "22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7"). InnerVolumeSpecName "kube-api-access-5mqlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:25:42.239408 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.239372    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca" pod "22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" (UID: "22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:25:42.253901 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.253869    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:42.254114 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.254097    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mqlc\" (UniqueName: \"kubernetes.io/projected/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-kube-api-access-5mqlc\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:42.254286 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.254269    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:42.483531 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.483494    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x" event={"ID":"22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7","Type":"ContainerDied","Data":"8b3f5ca2ea6b4a02a0590ab6c2a414a723bc35c045bf1affe65c9956e66bb9e1"}
Nov 03 15:25:42.484130 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.484108    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b3f5ca2ea6b4a02a0590ab6c2a414a723bc35c045bf1affe65c9956e66bb9e1"
Nov 03 15:25:42.484343 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:42.484327    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c681e757-vrp5x"
Nov 03 15:25:49.291070 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.291038    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"]
Nov 03 15:25:49.291584 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.291475    6481 cpu_manager.go:414] "containerMap: removing container" podUID="22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" containerName="revoke-keys"
Nov 03 15:25:49.291584 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.291488    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" containerName="revoke-keys"
Nov 03 15:25:49.291584 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.291585    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7" containerName="revoke-keys"
Nov 03 15:25:49.294716 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.294696    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.311424 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.311400    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:25:49.323204 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.323156    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:25:49.323204 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.323181    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:25:49.323458 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.323254    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:25:49.326036 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.326017    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"]
Nov 03 15:25:49.350687 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.350657    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ml7r\" (UniqueName: \"kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.350823 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.350695    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.350947 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.350921    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.451475 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.451437    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ml7r\" (UniqueName: \"kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.451702 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.451482    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.451702 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.451614    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.452288 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.452259    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.453875 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.453848    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.473747 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.473725    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ml7r\" (UniqueName: \"kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r\") pod \"maas-api-revoke-sub-a676f8ca-nh59l\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.607140 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.607080    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:49.905430 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:49.905405    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"]
Nov 03 15:25:49.907559 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:25:49.907527    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93b2ffe_3f05_41d8_9849_63f859effca7.slice/crio-3ea3de3fc481125156a4938c5a2a9128cb3d7cc47128d277c31b15377a0a0ced WatchSource:0}: Error finding container 3ea3de3fc481125156a4938c5a2a9128cb3d7cc47128d277c31b15377a0a0ced: Status 404 returned error can't find the container with id 3ea3de3fc481125156a4938c5a2a9128cb3d7cc47128d277c31b15377a0a0ced
Nov 03 15:25:50.552899 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:50.552812    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l" event={"ID":"e93b2ffe-3f05-41d8-9849-63f859effca7","Type":"ContainerStarted","Data":"dcc54e180b8e081a903b489672d37f4983e09e3a667761775a912659ae4f8e8b"}
Nov 03 15:25:50.552899 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:50.552877    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l" event={"ID":"e93b2ffe-3f05-41d8-9849-63f859effca7","Type":"ContainerStarted","Data":"3ea3de3fc481125156a4938c5a2a9128cb3d7cc47128d277c31b15377a0a0ced"}
Nov 03 15:25:50.577259 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:50.577208    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:25:50.608710 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:50.608667    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l" podStartSLOduration=1.6086542860000002 podStartE2EDuration="1.608654286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:25:50.60764639 +0000 UTC m=+4428.657677555" watchObservedRunningTime="2026-11-03 15:25:50.608654286 +0000 UTC m=+4428.658685468"
Nov 03 15:25:51.564599 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:51.564553    6481 generic.go:356] "Generic (PLEG): container finished" podID="e93b2ffe-3f05-41d8-9849-63f859effca7" containerID="dcc54e180b8e081a903b489672d37f4983e09e3a667761775a912659ae4f8e8b" exitCode=0
Nov 03 15:25:51.565131 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:51.564610    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l" event={"ID":"e93b2ffe-3f05-41d8-9849-63f859effca7","Type":"ContainerDied","Data":"dcc54e180b8e081a903b489672d37f4983e09e3a667761775a912659ae4f8e8b"}
Nov 03 15:25:53.238532 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.238258    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:53.383255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.377277    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca\") pod \"e93b2ffe-3f05-41d8-9849-63f859effca7\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") "
Nov 03 15:25:53.383255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.377331    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca\") pod \"e93b2ffe-3f05-41d8-9849-63f859effca7\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") "
Nov 03 15:25:53.383255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.377453    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r\" (UniqueName: \"kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r\") pod \"e93b2ffe-3f05-41d8-9849-63f859effca7\" (UID: \"e93b2ffe-3f05-41d8-9849-63f859effca7\") "
Nov 03 15:25:53.402902 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.402440    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca" pod "e93b2ffe-3f05-41d8-9849-63f859effca7" (UID: "e93b2ffe-3f05-41d8-9849-63f859effca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:25:53.410287 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.408375    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca" pod "e93b2ffe-3f05-41d8-9849-63f859effca7" (UID: "e93b2ffe-3f05-41d8-9849-63f859effca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:25:53.441678 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.437361    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r" pod "e93b2ffe-3f05-41d8-9849-63f859effca7" (UID: "e93b2ffe-3f05-41d8-9849-63f859effca7"). InnerVolumeSpecName "kube-api-access-6ml7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:25:53.486255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.480893    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:53.486255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.480928    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e93b2ffe-3f05-41d8-9849-63f859effca7-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:53.486255 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.480944    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ml7r\" (UniqueName: \"kubernetes.io/projected/e93b2ffe-3f05-41d8-9849-63f859effca7-kube-api-access-6ml7r\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:25:53.597251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.593683    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l" event={"ID":"e93b2ffe-3f05-41d8-9849-63f859effca7","Type":"ContainerDied","Data":"3ea3de3fc481125156a4938c5a2a9128cb3d7cc47128d277c31b15377a0a0ced"}
Nov 03 15:25:53.597251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.593754    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea3de3fc481125156a4938c5a2a9128cb3d7cc47128d277c31b15377a0a0ced"
Nov 03 15:25:53.597251 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:53.593823    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a676f8ca-nh59l"
Nov 03 15:25:57.985497 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:25:57.985463    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:25:59.719172 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.719138    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"]
Nov 03 15:25:59.719844 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.719819    6481 cpu_manager.go:414] "containerMap: removing container" podUID="e93b2ffe-3f05-41d8-9849-63f859effca7" containerName="revoke-keys"
Nov 03 15:25:59.719895 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.719849    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93b2ffe-3f05-41d8-9849-63f859effca7" containerName="revoke-keys"
Nov 03 15:25:59.720012 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.719998    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93b2ffe-3f05-41d8-9849-63f859effca7" containerName="revoke-keys"
Nov 03 15:25:59.723389 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.723373    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.746299 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.746271    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"]
Nov 03 15:25:59.754330 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.754305    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 15:25:59.754524 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.754505    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 15:25:59.754609 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.754562    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-srslx\""
Nov 03 15:25:59.754676 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.754655    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 15:25:59.781707 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.781672    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.781851 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.781825    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.781930 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.781861    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk2lw\" (UniqueName: \"kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.883316 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.883286    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.883480 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.883339    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.883525 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.883502    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk2lw\" (UniqueName: \"kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.883916 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.883883    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.885653 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.885631    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:25:59.904720 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:25:59.904695    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk2lw\" (UniqueName: \"kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw\") pod \"maas-api-revoke-sub-b3923311-8xv5f\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:26:00.036968 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:00.036881    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:26:00.340891 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:00.340859    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"]
Nov 03 15:26:00.341047 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:26:00.340912    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1191f0f0_b626_416e_b246_ea5e8a2dbeea.slice/crio-7de5812103e1e8a3bad7e0d7a26ddc6bf0048b6b8f5a4d0cf376c93d9ac3f429 WatchSource:0}: Error finding container 7de5812103e1e8a3bad7e0d7a26ddc6bf0048b6b8f5a4d0cf376c93d9ac3f429: Status 404 returned error can't find the container with id 7de5812103e1e8a3bad7e0d7a26ddc6bf0048b6b8f5a4d0cf376c93d9ac3f429
Nov 03 15:26:00.652316 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:00.652283    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f" event={"ID":"1191f0f0-b626-416e-b246-ea5e8a2dbeea","Type":"ContainerStarted","Data":"f37570a57ccb45080f1a71f81f00382a036ed6eca49e5e7bc6e637fed9363366"}
Nov 03 15:26:00.652455 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:00.652361    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f" event={"ID":"1191f0f0-b626-416e-b246-ea5e8a2dbeea","Type":"ContainerStarted","Data":"7de5812103e1e8a3bad7e0d7a26ddc6bf0048b6b8f5a4d0cf376c93d9ac3f429"}
Nov 03 15:26:00.697072 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:00.697024    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f" podStartSLOduration=1.697011189 podStartE2EDuration="1.697011189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:26:00.695261975 +0000 UTC m=+4438.745293159" watchObservedRunningTime="2026-11-03 15:26:00.697011189 +0000 UTC m=+4438.747042371"
Nov 03 15:26:01.689624 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:01.689493    6481 generic.go:356] "Generic (PLEG): container finished" podID="1191f0f0-b626-416e-b246-ea5e8a2dbeea" containerID="f37570a57ccb45080f1a71f81f00382a036ed6eca49e5e7bc6e637fed9363366" exitCode=0
Nov 03 15:26:01.689624 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:01.689557    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f" event={"ID":"1191f0f0-b626-416e-b246-ea5e8a2dbeea","Type":"ContainerDied","Data":"f37570a57ccb45080f1a71f81f00382a036ed6eca49e5e7bc6e637fed9363366"}
Nov 03 15:26:03.353962 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.353631    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:26:03.399599 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.399558    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca\") pod \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") "
Nov 03 15:26:03.400351 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.400326    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw\" (UniqueName: \"kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw\") pod \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") "
Nov 03 15:26:03.400630 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.400609    6481 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca\") pod \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\" (UID: \"1191f0f0-b626-416e-b246-ea5e8a2dbeea\") "
Nov 03 15:26:03.417254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.409713    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw" pod "1191f0f0-b626-416e-b246-ea5e8a2dbeea" (UID: "1191f0f0-b626-416e-b246-ea5e8a2dbeea"). InnerVolumeSpecName "kube-api-access-wk2lw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 15:26:03.417784 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.417757    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca" pod "1191f0f0-b626-416e-b246-ea5e8a2dbeea" (UID: "1191f0f0-b626-416e-b246-ea5e8a2dbeea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 15:26:03.418718 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.418694    6481 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca" pod "1191f0f0-b626-416e-b246-ea5e8a2dbeea" (UID: "1191f0f0-b626-416e-b246-ea5e8a2dbeea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 15:26:03.505307 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.502922    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-openshift-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:26:03.505307 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.502959    6481 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1191f0f0-b626-416e-b246-ea5e8a2dbeea-maas-api-xks-ca\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:26:03.505307 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.502975    6481 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk2lw\" (UniqueName: \"kubernetes.io/projected/1191f0f0-b626-416e-b246-ea5e8a2dbeea-kube-api-access-wk2lw\") on node \"ip-10-0-128-190.ec2.internal\" DevicePath \"\""
Nov 03 15:26:03.740258 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.732397    6481 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f"
Nov 03 15:26:03.740258 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.732614    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3923311-8xv5f" event={"ID":"1191f0f0-b626-416e-b246-ea5e8a2dbeea","Type":"ContainerDied","Data":"7de5812103e1e8a3bad7e0d7a26ddc6bf0048b6b8f5a4d0cf376c93d9ac3f429"}
Nov 03 15:26:03.740258 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:03.732762    6481 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7de5812103e1e8a3bad7e0d7a26ddc6bf0048b6b8f5a4d0cf376c93d9ac3f429"
Nov 03 15:26:05.228379 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:05.228342    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:26:10.219037 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:10.219005    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:26:10.446211 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:10.446179    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:26:11.077488 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:11.077457    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-f8vvn"]
Nov 03 15:26:21.480446 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:21.480415    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948677c5f-lc4kc_1a4df69a-9878-4445-a3ab-94d60af00096/authorino/0.log"
Nov 03 15:26:27.989091 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:26:27.989053    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:26:34.046178 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:34.046144    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"]
Nov 03 15:26:34.059269 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:34.059244    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81f076c9-dwccc"]
Nov 03 15:26:34.066171 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:34.066144    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b70daab9-b0f4-4389-9cd4-40e8cb05bbd1" path="/var/lib/kubelet/pods/b70daab9-b0f4-4389-9cd4-40e8cb05bbd1/volumes"
Nov 03 15:26:34.286432 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:34.286405    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78fsn_5e1d2c0b-5871-481c-83b2-48f62884699d/server/0.log"
Nov 03 15:26:34.466103 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:34.466073    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nmtcb_07e51ae4-b9c2-4424-b835-81b5b8f33934/manager/0.log"
Nov 03 15:26:34.691759 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:34.691731    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kjghr_e5dafb2a-ee70-4307-8977-f1289ea765aa/proxy/0.log"
Nov 03 15:26:36.057886 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:36.057856    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0013a6d7-z6fkg_87d47603-32f0-4ff8-ae44-eafa13ee3c90/revoke-keys/0.log"
Nov 03 15:26:36.241682 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:36.241652    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1145cec5-dlzk2_f5c3b90a-a636-4422-9bb5-2ed66b6896f0/revoke-keys/0.log"
Nov 03 15:26:36.421174 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:36.421150    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-26dfdc6d-zq8d9_6306b5f5-474d-4fa4-8b76-131f1b0ac88c/revoke-keys/0.log"
Nov 03 15:26:36.605011 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:36.604979    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-26f2e7aa-2kpjx_3686fec0-1b49-4ca6-9df0-7b769a8ead67/revoke-keys/0.log"
Nov 03 15:26:36.786254 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:36.786170    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2930fb36-wmwqb_462cc4fc-b9a3-4064-950d-e76a79333ba6/revoke-keys/0.log"
Nov 03 15:26:36.968075 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:36.968051    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3332baab-ngjhm_e77e2c37-353f-40b8-a7f2-48f51d5e151c/revoke-keys/0.log"
Nov 03 15:26:37.139420 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:37.139347    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3745d2b7-w8t5j_58aefa24-dc79-4d8d-8c62-c79e1829145a/revoke-keys/0.log"
Nov 03 15:26:37.308913 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:37.308887    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-48ed8ae3-9db2c_9024c37e-41aa-4f69-aa50-f6d31bc7ca8e/revoke-keys/0.log"
Nov 03 15:26:37.483564 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:37.483535    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-83bc83d0-wc2w7_f2f37139-c35a-4cbe-ac67-3cc7b25689d5/revoke-keys/0.log"
Nov 03 15:26:37.664894 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:37.664864    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-922c97e6-tlkc2_2fc6c93b-a854-41c5-a67a-9a0f878e2bc3/revoke-keys/0.log"
Nov 03 15:26:37.847137 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:37.847062    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-94f68e24-slzs6_e7e291e8-9e32-496f-bb17-4413d100335c/revoke-keys/0.log"
Nov 03 15:26:38.036024 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:38.036000    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9d8eeeb0-pv2nt_26535a8e-c190-49a8-8818-7866725b4d78/revoke-keys/0.log"
Nov 03 15:26:38.220750 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:38.220726    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a11c6fa9-z2gdf_8ea6a2af-3bf7-47c9-8d9a-10b1769453f5/revoke-keys/0.log"
Nov 03 15:26:38.401757 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:38.401728    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a676f8ca-nh59l_e93b2ffe-3f05-41d8-9849-63f859effca7/revoke-keys/0.log"
Nov 03 15:26:38.594003 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:38.593927    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b3923311-8xv5f_1191f0f0-b626-416e-b246-ea5e8a2dbeea/revoke-keys/0.log"
Nov 03 15:26:38.773984 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:38.773955    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c3e6c43c-xg4n7_20a991b3-f88a-40be-ab50-a9d3f337d902/revoke-keys/0.log"
Nov 03 15:26:38.955253 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:38.955212    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c681e757-vrp5x_22fa10d6-c11c-4d49-b2bc-7bdb9e633bd7/revoke-keys/0.log"
Nov 03 15:26:39.127249 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:39.127198    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d13d82ee-7pfd5_ff5164cf-509f-42ce-9fef-5df14a91aebb/revoke-keys/0.log"
Nov 03 15:26:41.156484 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:41.156450    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948677c5f-lc4kc_1a4df69a-9878-4445-a3ab-94d60af00096/authorino/0.log"
Nov 03 15:26:42.240421 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:42.240388    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8vvn_5135bcc0-45a6-414e-910c-f68bed8be41a/limitador/0.log"
Nov 03 15:26:43.409268 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:43.409218    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-jtx2p_2dfe3d02-9865-458c-9d3f-9b822be998dd/istio-proxy/0.log"
Nov 03 15:26:43.818110 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:43.818037    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76667dfcdc-6cmzw_29b46fcd-3054-4c92-ab92-d4811de95f03/kube-auth-proxy/0.log"
Nov 03 15:26:44.059600 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:44.059569    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"]
Nov 03 15:26:44.074365 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:44.074275    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94f68e24-slzs6"]
Nov 03 15:26:44.187908 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:44.187884    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-2p5f4_0b5ad832-aa94-4941-9b2e-852a9ddfda03/istio-proxy/0.log"
Nov 03 15:26:44.795473 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:44.795448    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-c8dts_1cca3427-0571-4cd1-82be-b1bbaaefc72f/istio-proxy/0.log"
Nov 03 15:26:45.773075 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:45.773048    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-wxgc4_8c1222d6-eef5-45c9-88ac-84c8b86a60d3/istio-proxy/0.log"
Nov 03 15:26:45.976080 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:45.976053    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-z6fd6_7285fd08-91c8-4113-9f66-d5e6810b02d5/istio-proxy/0.log"
Nov 03 15:26:46.066480 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:46.066404    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e291e8-9e32-496f-bb17-4413d100335c" path="/var/lib/kubelet/pods/e7e291e8-9e32-496f-bb17-4413d100335c/volumes"
Nov 03 15:26:46.156917 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:46.156889    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t_0862f426-a7ac-4f13-a39f-0e0925aa4c07/istio-proxy/0.log"
Nov 03 15:26:46.512313 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:46.512284    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-95b747dcd-rw869_3aff64b7-93ff-44f1-b495-3dab807c820d/payload-processing/0.log"
Nov 03 15:26:48.030812 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:48.030788    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-pbhb4_5b72d32b-da35-496d-a85e-327eac0da3f1/main/0.log"
Nov 03 15:26:48.216101 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:48.216077    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tr28v_9c143c15-a7c2-4d1f-8e60-216aae706edf/main/0.log"
Nov 03 15:26:48.976620 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:48.976590    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kt6ts_6212728d-6351-4e9b-bbfd-f9424c48412b/main/0.log"
Nov 03 15:26:57.985702 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:26:57.985659    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:26:58.878562 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.878525    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5zrvw/must-gather-ltvnx"]
Nov 03 15:26:58.879140 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.879116    6481 cpu_manager.go:414] "containerMap: removing container" podUID="1191f0f0-b626-416e-b246-ea5e8a2dbeea" containerName="revoke-keys"
Nov 03 15:26:58.879140 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.879139    6481 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1191f0f0-b626-416e-b246-ea5e8a2dbeea" containerName="revoke-keys"
Nov 03 15:26:58.879390 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.879297    6481 memory_manager.go:365] "RemoveStaleState removing state" podUID="1191f0f0-b626-416e-b246-ea5e8a2dbeea" containerName="revoke-keys"
Nov 03 15:26:58.883011 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.882990    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:58.904294 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.904272    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5zrvw/must-gather-ltvnx"]
Nov 03 15:26:58.913768 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.913744    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5zrvw\"/\"openshift-service-ca.crt\""
Nov 03 15:26:58.913888 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.913770    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5zrvw\"/\"default-dockercfg-sdb9v\""
Nov 03 15:26:58.913888 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.913836    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5zrvw\"/\"kube-root-ca.crt\""
Nov 03 15:26:58.953340 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.953317    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4752r\" (UniqueName: \"kubernetes.io/projected/9e2e8480-27c6-4c7e-ab0f-00b31353e3e1-kube-api-access-4752r\") pod \"must-gather-ltvnx\" (UID: \"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1\") " pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:58.953464 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.953446    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e2e8480-27c6-4c7e-ab0f-00b31353e3e1-must-gather-output\") pod \"must-gather-ltvnx\" (UID: \"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1\") " pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:58.974677 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:58.974650    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:26:59.054258 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.054216    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4752r\" (UniqueName: \"kubernetes.io/projected/9e2e8480-27c6-4c7e-ab0f-00b31353e3e1-kube-api-access-4752r\") pod \"must-gather-ltvnx\" (UID: \"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1\") " pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:59.054620 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.054363    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e2e8480-27c6-4c7e-ab0f-00b31353e3e1-must-gather-output\") pod \"must-gather-ltvnx\" (UID: \"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1\") " pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:59.054745 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.054724    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9e2e8480-27c6-4c7e-ab0f-00b31353e3e1-must-gather-output\") pod \"must-gather-ltvnx\" (UID: \"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1\") " pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:59.077181 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.077152    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4752r\" (UniqueName: \"kubernetes.io/projected/9e2e8480-27c6-4c7e-ab0f-00b31353e3e1-kube-api-access-4752r\") pod \"must-gather-ltvnx\" (UID: \"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1\") " pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:59.197313 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.197274    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5zrvw/must-gather-ltvnx"
Nov 03 15:26:59.198211 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.198186    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:26:59.500533 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.500505    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5zrvw/must-gather-ltvnx"]
Nov 03 15:26:59.500891 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:26:59.500869    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e2e8480_27c6_4c7e_ab0f_00b31353e3e1.slice/crio-e27ae3cdb1cfb9ce43691b4f05b243bf032a0424128e3e9d96ccf7e5fbbcd813 WatchSource:0}: Error finding container e27ae3cdb1cfb9ce43691b4f05b243bf032a0424128e3e9d96ccf7e5fbbcd813: Status 404 returned error can't find the container with id e27ae3cdb1cfb9ce43691b4f05b243bf032a0424128e3e9d96ccf7e5fbbcd813
Nov 03 15:26:59.502774 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:26:59.502757    6481 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 15:27:00.091111 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:00.091079    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5zrvw/must-gather-ltvnx" event={"ID":"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1","Type":"ContainerStarted","Data":"e27ae3cdb1cfb9ce43691b4f05b243bf032a0424128e3e9d96ccf7e5fbbcd813"}
Nov 03 15:27:03.799400 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:03.799374    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8flw8_9927b257-7027-4ef3-aa19-bde73b4d93b9/global-pull-secret-syncer/0.log"
Nov 03 15:27:04.608552 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:04.608503    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log"
Nov 03 15:27:05.099529 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:05.099501    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-190.ec2.internal_61a071342255b3898148bdccf890b4bc/haproxy/0.log"
Nov 03 15:27:05.339222 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:05.339170    6481 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7t76q_d6e29be6-c48c-4bc8-a601-a0e337715b61/konnectivity-agent/0.log" limit=1024
Nov 03 15:27:06.136667 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:06.136577    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5zrvw/must-gather-ltvnx" event={"ID":"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1","Type":"ContainerStarted","Data":"f96cd8c63cbe31ef0047f688f36e705128930a9ec9fb0aeade20ed3df169cb29"}
Nov 03 15:27:06.136667 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:06.136627    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5zrvw/must-gather-ltvnx" event={"ID":"9e2e8480-27c6-4c7e-ab0f-00b31353e3e1","Type":"ContainerStarted","Data":"6091fb3f7b8213738d797a8742a92f64bcb629bfc67fb327df2793939450e138"}
Nov 03 15:27:06.180775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:06.180728    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5zrvw/must-gather-ltvnx" podStartSLOduration=1.996690293 podStartE2EDuration="8.180712444s" totalImagesPullingTime="6.184022151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:06.180035183 +0000 UTC m=+4504.230066367" watchObservedRunningTime="2026-11-03 15:27:06.180712444 +0000 UTC m=+4504.230743628"
Nov 03 15:27:08.810729 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:08.810697    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78fsn_5e1d2c0b-5871-481c-83b2-48f62884699d/server/0.log"
Nov 03 15:27:09.528353 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:09.528325    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nmtcb_07e51ae4-b9c2-4424-b835-81b5b8f33934/manager/0.log"
Nov 03 15:27:09.570613 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:09.570585    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kjghr_e5dafb2a-ee70-4307-8977-f1289ea765aa/proxy/0.log"
Nov 03 15:27:10.070643 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:10.070609    6481 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"]
Nov 03 15:27:10.083648 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:10.083620    6481 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3745d2b7-w8t5j"]
Nov 03 15:27:10.768943 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:10.768909    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948677c5f-lc4kc_1a4df69a-9878-4445-a3ab-94d60af00096/authorino/0.log"
Nov 03 15:27:12.065198 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:12.065162    6481 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aefa24-dc79-4d8d-8c62-c79e1829145a" path="/var/lib/kubelet/pods/58aefa24-dc79-4d8d-8c62-c79e1829145a/volumes"
Nov 03 15:27:12.225660 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:12.225630    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78fsn_5e1d2c0b-5871-481c-83b2-48f62884699d/server/0.log"
Nov 03 15:27:12.247711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:12.247686    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8vvn_5135bcc0-45a6-414e-910c-f68bed8be41a/limitador/0.log"
Nov 03 15:27:12.948482 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:12.948442    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nmtcb_07e51ae4-b9c2-4424-b835-81b5b8f33934/manager/0.log"
Nov 03 15:27:13.052489 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.052420    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kjghr_e5dafb2a-ee70-4307-8977-f1289ea765aa/proxy/0.log"
Nov 03 15:27:13.810053 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.810016    6481 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"]
Nov 03 15:27:13.814058 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.814041    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:13.841257 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.841212    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"]
Nov 03 15:27:13.847940 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.847917    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rfl4d\"/\"kube-root-ca.crt\""
Nov 03 15:27:13.848071 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.847974    6481 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rfl4d\"/\"openshift-service-ca.crt\""
Nov 03 15:27:13.911101 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.911072    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-proc\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:13.911298 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.911133    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-sys\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:13.911371 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.911311    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-podres\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:13.911445 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.911425    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mghrp\" (UniqueName: \"kubernetes.io/projected/bdfa2261-38b3-4213-8b3f-151c47be23d1-kube-api-access-mghrp\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:13.911508 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.911475    6481 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-lib-modules\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:13.916176 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:13.916156    6481 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rfl4d\"/\"default-dockercfg-nh78f\""
Nov 03 15:27:14.013076 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013045    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-lib-modules\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013086    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-proc\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013148    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-sys\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013205    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-lib-modules\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013253    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-proc\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013283 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013277    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-podres\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013477 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013292    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-sys\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013477 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013315    6481 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mghrp\" (UniqueName: \"kubernetes.io/projected/bdfa2261-38b3-4213-8b3f-151c47be23d1-kube-api-access-mghrp\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.013477 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.013368    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bdfa2261-38b3-4213-8b3f-151c47be23d1-podres\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.038036 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.038010    6481 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mghrp\" (UniqueName: \"kubernetes.io/projected/bdfa2261-38b3-4213-8b3f-151c47be23d1-kube-api-access-mghrp\") pod \"perf-node-gather-daemonset-9w2vf\" (UID: \"bdfa2261-38b3-4213-8b3f-151c47be23d1\") " pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.104833 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.104773    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/alertmanager/0.log"
Nov 03 15:27:14.125645 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.125619    6481 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:14.245761 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.245732    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/config-reloader/0.log"
Nov 03 15:27:14.388766 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.388681    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/kube-rbac-proxy-web/0.log"
Nov 03 15:27:14.439573 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.439538    6481 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"]
Nov 03 15:27:14.440374 ip-10-0-128-190 kubenswrapper[6481]: W1103 15:27:14.440342    6481 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbdfa2261_38b3_4213_8b3f_151c47be23d1.slice/crio-ee2b42e23e9c28f3bcfebad72e193364d2788eea5fc77f8dd69bb679b362a0bf WatchSource:0}: Error finding container ee2b42e23e9c28f3bcfebad72e193364d2788eea5fc77f8dd69bb679b362a0bf: Status 404 returned error can't find the container with id ee2b42e23e9c28f3bcfebad72e193364d2788eea5fc77f8dd69bb679b362a0bf
Nov 03 15:27:14.543378 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.543349    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/kube-rbac-proxy/0.log"
Nov 03 15:27:14.684991 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.684951    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/kube-rbac-proxy-metric/0.log"
Nov 03 15:27:14.824743 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.824714    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/prom-label-proxy/0.log"
Nov 03 15:27:14.970097 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:14.970016    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d49b0b8-b06b-4a64-8f1d-b1f0a6951749/init-config-reloader/0.log"
Nov 03 15:27:15.217860 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:15.217823    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf" event={"ID":"bdfa2261-38b3-4213-8b3f-151c47be23d1","Type":"ContainerStarted","Data":"80097f682af8186f2595bf5f6308db3e6ed2232b7403dfd790e828f1b1955e99"}
Nov 03 15:27:15.218056 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:15.217877    6481 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf" event={"ID":"bdfa2261-38b3-4213-8b3f-151c47be23d1","Type":"ContainerStarted","Data":"ee2b42e23e9c28f3bcfebad72e193364d2788eea5fc77f8dd69bb679b362a0bf"}
Nov 03 15:27:15.218056 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:15.217925    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:15.530936 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:15.530906    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-78fsn_5e1d2c0b-5871-481c-83b2-48f62884699d/server/0.log"
Nov 03 15:27:15.738238 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:15.738211    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6679c484c8-thcsr_0bc6d4db-06d7-4767-bef1-d90cfb51dc88/metrics-server/0.log"
Nov 03 15:27:15.894710 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:15.894627    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-c2pfl_9225fde2-c7af-4b4e-80c0-6a473f655fec/monitoring-plugin/0.log"
Nov 03 15:27:16.075405 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:16.075373    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n87hh_e3ccb6ab-b665-4753-96a2-9ce1fa145666/node-exporter/0.log"
Nov 03 15:27:16.220775 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:16.220745    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n87hh_e3ccb6ab-b665-4753-96a2-9ce1fa145666/kube-rbac-proxy/0.log"
Nov 03 15:27:16.260090 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:16.260062    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-nmtcb_07e51ae4-b9c2-4424-b835-81b5b8f33934/manager/0.log"
Nov 03 15:27:16.332711 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:16.332680    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-kjghr_e5dafb2a-ee70-4307-8977-f1289ea765aa/proxy/0.log"
Nov 03 15:27:16.371670 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:16.371644    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n87hh_e3ccb6ab-b665-4753-96a2-9ce1fa145666/init-textfile/0.log"
Nov 03 15:27:17.819524 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:17.819489    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/prometheus/0.log"
Nov 03 15:27:17.972724 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:17.972700    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/config-reloader/0.log"
Nov 03 15:27:18.038502 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:18.038464    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-jtx2p_2dfe3d02-9865-458c-9d3f-9b822be998dd/istio-proxy/0.log"
Nov 03 15:27:18.112344 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:18.112272    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/thanos-sidecar/0.log"
Nov 03 15:27:18.263319 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:18.263293    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/kube-rbac-proxy-web/0.log"
Nov 03 15:27:18.417571 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:18.417524    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/kube-rbac-proxy/0.log"
Nov 03 15:27:18.568725 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:18.568695    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/kube-rbac-proxy-thanos/0.log"
Nov 03 15:27:18.728917 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:18.728852    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4e9d1a17-5793-4dfe-b7d3-b61f8a12ecd5/init-config-reloader/0.log"
Nov 03 15:27:19.825980 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:19.825952    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f8554fdc-w8wsw_f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/thanos-query/0.log"
Nov 03 15:27:19.970154 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:19.970126    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f8554fdc-w8wsw_f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/kube-rbac-proxy-web/0.log"
Nov 03 15:27:20.118399 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:20.118331    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f8554fdc-w8wsw_f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/kube-rbac-proxy/0.log"
Nov 03 15:27:20.265039 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:20.265009    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f8554fdc-w8wsw_f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/prom-label-proxy/0.log"
Nov 03 15:27:20.412994 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:20.412970    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f8554fdc-w8wsw_f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/kube-rbac-proxy-rules/0.log"
Nov 03 15:27:20.566105 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:20.566079    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-76f8554fdc-w8wsw_f983b9ee-1e60-4ec4-a196-e40be0d0ccd9/kube-rbac-proxy-metrics/0.log"
Nov 03 15:27:21.173369 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:21.173343    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76667dfcdc-6cmzw_29b46fcd-3054-4c92-ab92-d4811de95f03/kube-auth-proxy/0.log"
Nov 03 15:27:21.234988 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:21.234963    6481 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf"
Nov 03 15:27:21.281675 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:21.281631    6481 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rfl4d/perf-node-gather-daemonset-9w2vf" podStartSLOduration=8.281617672 podStartE2EDuration="8.281617672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:27:15.269486515 +0000 UTC m=+4513.319517702" watchObservedRunningTime="2026-11-03 15:27:21.281617672 +0000 UTC m=+4519.331648900"
Nov 03 15:27:21.325366 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:21.325341    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-2p5f4_0b5ad832-aa94-4941-9b2e-852a9ddfda03/istio-proxy/0.log"
Nov 03 15:27:21.552840 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:21.552770    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-c8dts_1cca3427-0571-4cd1-82be-b1bbaaefc72f/istio-proxy/0.log"
Nov 03 15:27:21.939703 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:21.939668    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-wxgc4_8c1222d6-eef5-45c9-88ac-84c8b86a60d3/istio-proxy/0.log"
Nov 03 15:27:22.014484 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:22.014456    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-z6fd6_7285fd08-91c8-4113-9f66-d5e6810b02d5/istio-proxy/0.log"
Nov 03 15:27:22.094500 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:22.094474    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t_0862f426-a7ac-4f13-a39f-0e0925aa4c07/istio-proxy/0.log"
Nov 03 15:27:22.244953 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:22.244880    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-95b747dcd-rw869_3aff64b7-93ff-44f1-b495-3dab807c820d/payload-processing/0.log"
Nov 03 15:27:25.081536 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:25.081504    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-948677c5f-lc4kc_1a4df69a-9878-4445-a3ab-94d60af00096/authorino/0.log"
Nov 03 15:27:25.565593 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:25.565570    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-f8vvn_5135bcc0-45a6-414e-910c-f68bed8be41a/limitador/0.log"
Nov 03 15:27:27.810914 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:27.810884    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-85d774877-jtx2p_2dfe3d02-9865-458c-9d3f-9b822be998dd/istio-proxy/0.log"
Nov 03 15:27:27.989637 ip-10-0-128-190 kubenswrapper[6481]: E1103 15:27:27.989596    6481 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kws78" containerName="extractor"
Nov 03 15:27:29.364733 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:29.364702    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fwrl7_3196f084-b0f3-4145-be7f-0d2b63dd8019/dns/0.log"
Nov 03 15:27:29.505647 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:29.505611    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fwrl7_3196f084-b0f3-4145-be7f-0d2b63dd8019/kube-rbac-proxy/0.log"
Nov 03 15:27:29.948585 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:29.948552    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g9r7l_b1c76fe7-8625-4a41-a039-af53c4978a94/dns-node-resolver/0.log"
Nov 03 15:27:31.069553 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:31.069513    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-76667dfcdc-6cmzw_29b46fcd-3054-4c92-ab92-d4811de95f03/kube-auth-proxy/0.log"
Nov 03 15:27:31.223632 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:31.223608    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-2p5f4_0b5ad832-aa94-4941-9b2e-852a9ddfda03/istio-proxy/0.log"
Nov 03 15:27:31.451332 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:31.451302    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-c8dts_1cca3427-0571-4cd1-82be-b1bbaaefc72f/istio-proxy/0.log"
Nov 03 15:27:31.758044 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:31.757971    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-868d4_8edbb9a8-e48f-428f-9967-61dd8333133e/node-ca/0.log"
Nov 03 15:27:31.856769 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:31.856738    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-wxgc4_8c1222d6-eef5-45c9-88ac-84c8b86a60d3/istio-proxy/0.log"
Nov 03 15:27:31.935002 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:31.934972    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-z6fd6_7285fd08-91c8-4113-9f66-d5e6810b02d5/istio-proxy/0.log"
Nov 03 15:27:32.007426 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:32.007399    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-6bd54bbbf9-mdj9t_0862f426-a7ac-4f13-a39f-0e0925aa4c07/istio-proxy/0.log"
Nov 03 15:27:32.151892 ip-10-0-128-190 kubenswrapper[6481]: I1103 15:27:32.151806    6481 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-95b747dcd-rw869_3aff64b7-93ff-44f1-b495-3dab807c820d/payload-processing/0.log"